Inhaltsverzeichnis von Unix Power Tools

Inhaltsvorschau

If we were writing about any other operating system, "power tools" might mean "nifty add-on utilities to extend the power of your operating system." That sounds suspiciously like a definition of Unix: an operating system loaded with decades' worth of nifty add-on utilities.

Unix is unique in that it wasn't designed as a commercial operating system meant to run application programs, but as a hacker's toolset, by and for programmers. In fact, an early release of the operating system went by the name PWB (Programmer's Work Bench).

When Ken Thompson and Dennis Ritchie first wrote Unix at AT&T Bell Labs, it was for their own use and for their friends and coworkers. Utility programs were added by various people as they had problems to solve. Because Bell Labs wasn't in the computer business, source code was given out to universities for a nominal fee. Brilliant researchers wrote their own software and added it to Unix in a spree of creative anarchy, which has been equaled only with Linux, in the introduction of the X Window System (Section 1.22), and especially the blend of Mac and Unix with Darwin included in the Mac OS X.

Unlike most other operating systems, where free software remains an unsupported add-on, Unix has taken as its own the work of thousands of independent programmers. During the commercialization of Unix within the past several years, this incorporation of outside software has slowed down for larger Unix installations, such as Sun's Solaris and HP's hp-ux, but not stopped entirely. This is especially true with the newer lighter versions of Unix, such as the various flavors of Linux and Darwin.

Therefore, a book on Unix inevitably has to focus not just on add-on utilities (though we do include many of those), but on how to use clever features of the many utilities that have been made part of Unix over the years.

Unix is also important to power users because it's one of the last popular operating systems that doesn't force you to work behind an interface of menus, windows, and mouse with a "one-size(-doesn't)-fit-all" programming interface. Yes, you can use Unix interfaces with windows and menus — and they can be great time savers in a lot of cases. But Unix also gives you building blocks that, with some training and practice, will give you many more choices than any software designer can cram onto a set of menus. If you learn to use Unix and its utilities from the command line, you don't have to be a programmer to do very powerful things with a few keystrokes.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If we were writing about any other operating system, "power tools" might mean "nifty add-on utilities to extend the power of your operating system." That sounds suspiciously like a definition of Unix: an operating system loaded with decades' worth of nifty add-on utilities.

Unix is unique in that it wasn't designed as a commercial operating system meant to run application programs, but as a hacker's toolset, by and for programmers. In fact, an early release of the operating system went by the name PWB (Programmer's Work Bench).

When Ken Thompson and Dennis Ritchie first wrote Unix at AT&T Bell Labs, it was for their own use and for their friends and coworkers. Utility programs were added by various people as they had problems to solve. Because Bell Labs wasn't in the computer business, source code was given out to universities for a nominal fee. Brilliant researchers wrote their own software and added it to Unix in a spree of creative anarchy, which has been equaled only with Linux, in the introduction of the X Window System (Section 1.22), and especially the blend of Mac and Unix with Darwin included in the Mac OS X.

Unlike most other operating systems, where free software remains an unsupported add-on, Unix has taken as its own the work of thousands of independent programmers. During the commercialization of Unix within the past several years, this incorporation of outside software has slowed down for larger Unix installations, such as Sun's Solaris and HP's hp-ux, but not stopped entirely. This is especially true with the newer lighter versions of Unix, such as the various flavors of Linux and Darwin.

Therefore, a book on Unix inevitably has to focus not just on add-on utilities (though we do include many of those), but on how to use clever features of the many utilities that have been made part of Unix over the years.

Unix is also important to power users because it's one of the last popular operating systems that doesn't force you to work behind an interface of menus, windows, and mouse with a "one-size(-doesn't)-fit-all" programming interface. Yes, you can use Unix interfaces with windows and menus — and they can be great time savers in a lot of cases. But Unix also gives you building blocks that, with some training and practice, will give you many more choices than any software designer can cram onto a set of menus. If you learn to use Unix and its utilities from the command line, you don't have to be a programmer to do very powerful things with a few keystrokes.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It has been said that Unix is not an operating system as much as it is a way of thinking. In The UNIX Programming Environment, Kernighan and Pike write that at the heart of the Unix philosophy "is the idea that the power of a system comes more from the relationships among programs than from the programs themselves."

Most of the nongraphical utility programs that have run under Unix since the beginning, some 30 years ago, share the same user interface. It's a minimal interface, to be sure — but one that allows programs to be strung together in pipelines to do jobs that no single program could do alone.

Most operating systems — including modern Unix and Linux systems — have graphical interfaces that are powerful and a pleasure to use. But none of them are so powerful or exciting to use as classic Unix pipes and filters, and the programming power of the shell.

A new user starts by stringing together simple pipelines and, when they get long enough, saving them for later execution in a file (Section 1.8), alias (Section 29.2), or function (Section 29.11). Gradually, if the user has the right temperament, he gets the idea that the computer can do more of the boring part of many jobs. Perhaps he starts out with a for loop (Section 28.9) to apply the same editing script to a series of files. Conditions and cases soon follow and before long, he finds himself programming.

On most systems, you need to learn consciously how to program. You must take up the study of one or more programming languages and expend a fair amount of concentrated effort before you can do anything productive. Unix, on the other hand, teaches programming imperceptibly — it is a slow but steady extension of the work you do simply by interacting with the computer.

Before long, you can step outside the bounds of the tools that have already been provided by the designers of the system and solve problems that don't quite fit the mold. This is sometimes called hacking; in other contexts, it is called "engineering." In essence, it is the ability to build a tool when the right one is not already on hand.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In recent times, more attention has been paid on the newer and more lightweight varieties of Unix: FreeBSD, Linux, and now Darwin — the version of BSD Unix that Apple used as the platform for the new Mac OS X. If you've worked with the larger Unix versions, you might be curious to see how it differs within these new environments.

For the most part, basic Unix functionality differs very little between implementations. For instance, I've not worked with a Unix box that doesn't have vi (Section 21.7) installed. Additionally, I've also not found any Unix system that doesn't have basic functionality, such as traversing directories with cd (Section 1.16) or getting additional help with man (Section 2.1).

However, what can differ between flavors of Unix is the behavior of some of the utilities and built-in commands, as well as the options. Even within a specific Unix flavor, such as FreeBSD, installations can differ because one installation uses the built-in version of a utility such as make (Section 40.3) and another installation has a GNU version of the same application.

An attempt was made to create some form of standardization with the POSIX effort. POSIX, which stands for Portable Operating System Interface, is an IEEE standard to work towards application interoperability. With this, C programs written on one flavor of Unix should work, with minimum modification, on another flavor of Unix.

Unfortunately, though the POSIX effort has had some impact on interoperability, there still are significant differences between Unix versions. In particular, something such as System V Unix can differ considerably from something such as Darwin.

However, there is stability in this seeming chaos: for the most part, the basic Unix utilities and commands behave the same in all Unix flavors, and aside from some optional differences, how a command works within one environment is exactly the same as in another environment. And if there are differences, using the facilities described in Chapter 2 should help you resolve these quickly.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Probably the single most important concept for would-be power users to grasp is that you don't "talk" directly to the Unix operating system. Instead, you talk to a program — and that program either talks to Unix itself or it talks to another program that talks to Unix. (When we say "talk" here, we mean communication using a keyboard and a mouse.)

There are three general kinds of programs you'll probably "talk" to:

The program called the shell (Section 27.1). A shell is a command interpreter. Its main job is to interpret the commands you type and to run the programs you specify in your command lines. By default, the shell reads commands from your tty and arranges for other programs to write their results there. The shell protects Unix from the user (and the user from Unix). It's the main focus of this book (and the rest of this article).
An interactive command, running "inside" a tty, that reads what you type directly. These take input directly from the user, without intervention from the shell. The shell's only job is to start them up. A text editor, a mail program, or almost any application program (such as word processing) includes its own command interpreter with its own rules. This book covers a few interactive commands — such as the vi editor — but its main focus is the shell and "noninteractive" utilities that the shell coordinates to do what needs doing.
A Graphical User Interface (GUI) with a desktop, windows, and so on. On Unix, a GUI is implemented with a set of running programs (all of which talk to Unix for you).
Unix was around long before GUIs were common, and there's no need to use a GUI to use Unix. In fact, Unix started in the days of teletypes, those clattering printing devices used to send telegrams. Unix terminals are still referred to as teletypes or ttys (Section 2.7).

The core of the Unix operating system is referred to as

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As pointed out by Kernighan and Pike in The UNIX Programming Environment, there are a number of principles that distinguish the Unix environment. One key concept is that programs are tools. Like all good tools, they should be specific in function, but usable for many different purposes.

In order for programs to become general-purpose tools, they must be data independent. This means three things:

Within limits, the output of any program should be usable as the input to another.
All of the information needed by a program should be either contained in the data stream passed to it or specified on the command line. A program should not prompt for input or do unnecessary formatting of output. In most cases, this means that Unix programs work with plain text files that don't contain "nonprintable" or "control" characters.
If no arguments are given, a program should read the standard input (usually the terminal keyboard) and write the standard output (usually the terminal screen).

Programs that can be used in this way are often called filters.

One of the most important consequences of these guidelines is that programs can be strung together in "pipelines" in which the output of one program is used as the input of another. A vertical bar (|) represents pipe and means "take the output of the program on the left and feed it into the program on the right."

For example, you can pipe the output of a search program to another program that sorts the output, and then pipe the result to the printer program or redirect it to a file (Section 43.1).

Not all Unix programs work together in this way. An interactive program like the Emacs editor (Section 19.1) generally doesn't read from or write to pipes you'd create on the command line. Instead, once the shell has started Emacs, the editor works independently of the shell (Section 1.4), reading its input and output directly from the terminal. And there are even exceptions to this exception. A program like less

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

With most operating systems, the command intepreter is built in; it is an integral part of the operating system. With Unix, your command interpreter is just another program. Traditionally, a command interpreter is called a "shell," perhaps because it protects you from the underlying kernel — or because it protects the kernel from you!

In the early 1980s, the most common shells were the Bourne shell (sh) and the C shell (csh). The Bourne shell (Section 3.3) (named after its creator, Steve Bourne) came first. It was excellent for shell programming (Section 1.8). But many Unix users (who were also writing programs in the C language) wanted a more familiar programming syntax — as well as more features for interactive use. So the C shell came from Berkeley as part of their Unix implementation. Soon (on systems that gave you the choice, at least) csh was much more popular for interactive use than sh. The C shell had a lot of nice features that weren't available in the original Bourne shell, including job control (Section 23.1) and history (Section 30.2). However, it wasn't hard for a shell programmer or an advanced user to push the C shell to its limits.

The Korn shell (also named after its creator, David Korn) arrived in the mid-1980s. The ksh is compatible with the Bourne shell, but has most of the C shell's features plus features like history editing (Section 30.14), often called command-line editing. The Korn shell was available only with a proprietary version of Unix, System V — but now a public-domain version named pdksh is widely available.

These days, most original C shell users have probably switched to tcsh (pronounced "T-shell"). It has all the features of csh and more — as well as fewer mis-features and outright bugs.

The "Bourne-again" shell, bash, is from the Free Software Foundation. It's fairly similar to the Korn shell. It has most of the C shell's features, plus command-line editing and a built-in help command. The programming syntax, though, is much more like the original Bourne shell — and many systems (including Linux) use

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can usually tell which family your shell belongs to by a character in the prompt it displays. Bourne-type shells, such as bash , usually have $ in the prompt. The C shell uses % (but tcsh users often use >).

If your shell has superuser (Section 1.18) privileges, though, the prompt typically ends with a hash, #.

To check the shell that runs automatically when you log in to Unix, type one of these commands (the second is for systems that use NIS, Sun's Network Information Service, to manage network-wide files):

% grep

               yourloginname /etc/passwd

% ypmatch 

               yourloginname passwd

You should get back the contents of your entry in the system password file. For example:

shelleyp:*:1006:1006:Shelley Powers:/usr/home/shelleyp:/usr/local/bin/bash

The fields are separated by colons, and the default shell is usually specified in the last field.

Note that in Mac OS X, passwords are managed and stored in Netinfo by default. To store the passwords in /etc/passwd, you'll need to configure this using Netinfo.

—TOR and SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the really wonderful things about the shell is that it doesn't just read and execute the commands you type at a prompt. The shell is a complete programming language.

The ease of shell programming is one of the real highlights of Unix for novices. A shell program need be no more than a single complex command line saved in a file — or a series of commands.

For example, let's say that you occasionally need to convert a Macintosh Microsoft Word file for use on your Unix system. Word lets you save the file in ASCII format. But there's a catch: the Mac uses a carriage return ASCII character 015 to mark the end of each line, while Unix uses a linefeed (ASCII 012). As a result, with Unix, the file looks like one long paragraph, with no end in sight.

That's easy to fix: the Unix tr (Section 21.11) command can convert every occurrence of one character in a file to another:

bash-2.04$ tr '\015' '\012' < 

               file.mac

                > 

               file.unix

But you're a novice, and you don't want to remember this particular piece of magic. Fine. Save the first part of this command line in a file called mac2unix in your personal bin directory (Section 7.4):

tr '\015' '\012'

Make the file executable with chmod (Section 50.5):

bash-2.04$ chmod +x mac2unix

Now you can say:

bash-2.04$ mac2unix < 

               file.mac

                > 

               file.unix

But why settle for that? What if you want to convert a bunch of files at once? Easy. The shell includes a general way of referring to arguments passed to a script and a number of looping constructs. The script:

for Section 35.21, $x Section 35.9

for x

do

    echo "Converting $x"

    tr '\015' '\012' < "$x" > "tmp.$x"

    mv "tmp.$x" "$x"

done

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some commands that you type are internal, which means they are built into the shell, and it's the shell that performs the action. For example, the cd command is built-in. The ls command, on the other hand, is an external program stored in the file /bin/ls.

The shell doesn't start a separate process to run internal commands. External commands require the shell to fork and exec (Section 27.2) a new subprocess (Section 24.3); this takes some time, especially on a busy system.

When you type the name of a command, the shell first checks to see if it is a built-in command and, if so, executes it. If the command name is an absolute pathname ( Section 1.16) beginning with /, like /bin/ls, there is no problem: the command is likewise executed. If the command is neither built-in nor specified with an absolute pathname, most shells (except the original Bourne shell) will check for aliases (Section 29.2) or shell functions (Section 29.11), which may have been defined by the user — often in a shell setup file (Section 3.3) that was read when the shell started. Most shells also "remember" the location of external commands (Section 27.6); this saves a long hunt down the search path. Finally, all shells look in the search path for an executable program or script with the given name.

The search path is exactly what its name implies: a list of directories that the shell should look through for a command whose name matches what is typed.

The search path isn't built into the shell; it's something you specify in your shell setup files.

By tradition, Unix system programs are kept in directories called /bin and /usr/bin, with additional programs usually used only by system administrators in either /etc and /usr/etc or /sbin and /usr/sbin. Many versions of Unix also have programs stored in /usr/ucb (named after the University of California at Berkeley, where many Unix programs were written). There may be other directories containing programs. For example, the programs that make up the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you have arrived at Unix via Windows 2000 or some other personal computer operating system, you will notice some big differences. Unix was, is, and always will be a multiuser operating system. It is a multiuser operating system even when you're the only person using it; it is a multiuser operating system even when it is running on a PC with a single keyboard; and this fact has important ramifications for everything that you do.

Why does this make a difference? Well, for one thing, you're never the only one using the system, even when you think you are. Don't bother to look under your desk to see if there's an extra terminal hidden down there. There isn't. But Unix is always doing things "behind your back," running programs of its own, whether you are aware of it or not. The most important of these programs, the kernel, is the heart of the Unix operating system itself. The kernel assigns memory to each of the programs that are running, partitions time fairly so that each program can get its job done, handles all I/O (input/output) operations, and so on. Another important group of programs, called daemons, are the system's "helpers." They run continuously — or from time to time — performing small but important tasks like handling mail, running network communications, feeding data to your printer, keeping track of the time, and so on.

Not only are you sharing the computer with the kernel and some mysterious daemons, you're also sharing it with yourself. You can issue the ps x (Section 24.5) command to get a list of all processes running on your system. For example:

  PID TTY    STAT  TIME COMMAND

18034 tty2   S     0:00 -zsh

18059 ?      S     0:01 ssh-agent

18088 tty2   S     0:00 sh /usr/X11R6/bin/startx

18096 tty2   S     0:00 xinit /etc/X11/xinit/xinitrc -- :0 -auth /home/jpeek/

18101 tty2   S     0:00 /usr/bin/gnome-session

18123 tty2   S     0:33 enlightenment -clientId default2

18127 tty2   S     0:01 magicdev --sm-client-id=default12

18141 tty2   S     0:03 panel --sm-client-id default8

18145 tty2   S     0:01 gmc --sm-client-id default10

18166 ?      S     1:20 gnomepager_applet --activate-goad-server gnomepager_a

18172 tty2   S     0:01 gnome-terminal

18174 tty2   S     0:00 gnome-pty-helper

18175 pts/0  S     0:00 zsh

18202 tty2   S     0:49 gnome-terminal

18203 tty2   S     0:00 gnome-pty-helper

18204 pts/1  S     0:01 zsh

18427 pts/1  T     0:00 man zshjp

18428 pts/1  T     0:00 sh -c /bin/gunzip -c /home/jpeek/.man/cat1/zshjp.1.gz

18430 pts/1  T     0:03 /usr/bin/less -is

18914 pts/1  T     0:02 vi upt3_changes.html

 1263 pts/1  T     0:00 vi urls.html

 1511 pts/1  T     0:00 less coding

 3363 pts/1  S     0:00 vi 1007.sgm

 4844 tty2   S     0:24 /usr/lib/netscape/netscape-communicator -irix-session

 4860 tty2   S     0:00 (dns helper)

 5055 pts/1  R     0:00 ps x

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Like all operating systems, Unix files have names. (Unix directories, devices, and so on also have filenames — and are treated like files (Section 1.19).) The names are words (sequences of characters) that let you identify a file. Older versions of Unix had some restrictions on the length of a filename (14 characters), but modern versions have removed these restrictions for all practical purposes. Sooner or later you will run into a limit, but if so, you are probably being unnecessarily verbose.

Technically, a filename can be made from almost any group of characters (including nonprinting characters and numbers) except a slash (/). However, you should avoid filenames containing most punctuation marks and all nonprinting characters. To be safe, limit your filenames to the following characters:

Upper- and lowercase characters: Unix filenames are always case sensitive. That is, upper- and lowercase letters are always different (unlike Microsoft Windows and others that consider upper- and lowercase letters the same). Therefore, myfile and Myfile are different files. It is usually a bad idea to have files whose names differ only in their capitalization, but that's your decision.
Underscores (_): Underscores are handy for separating "words" in a filename to make them more readable. For example, my_long_filename is easier to read than mylongfilename.
Periods (.): Periods are used by some programs (such as the C compiler) to separate filenames from filename extensions (Section 1.12). Extensions are used by these programs to recognize the type of file to be processed, but they are not treated specially by the shell, the kernel, or other Unix programs.
Filenames that begin with a period are treated specially by the shell: wildcards won't match (Section 1.13) them unless you include the period (like .*). The ls command, which lists your files, ignores files whose names begin with a period unless you give it a special option (ls -a (Section 8.9)). Special configuration files are often "hidden" in directories by beginning their names with a period.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In Microsoft Windows and some other operating systems, filenames often have the form name.extension. For example, plain text files have extensions such as .txt. The operating system treats the extension as separate from the filename and has rules about how long it must be, and so forth.

Unix doesn't have any special rules about extensions. The dot has no special meaning as a separator, and extensions can be any length. However, a number of programs (especially compilers) make use of extensions to recognize the different types of files they work with. In addition, there are a number of conventions that users have adopted to make clear the contents of their files. For example, you might name a text file containing some design notes notes.txt.

Table 1-1 lists some of the filename extensions you might see and a brief description of the programs that recognize them.

Table 1-1: Filename extensions that programs expect
Extension	Description
.a	Archive file (library)
.c	C program source file
.f	FORTRAN program source file
.F	FORTRAN program source file to preprocess
.gz	gzip ped file (Section 15.6)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The shells provide a number of wildcards that you can use to abbreviate filenames or refer to groups of files. For example, let's say you want to delete all filenames ending in .txt in the current directory (Section 1.16). You could delete these files one by one, but that would be boring if there were only 5 and very boring if there were 100. Instead, you can use a wildcarded name to say, "I want all files whose names end with .txt, regardless of what the first part is." The wildcard is the "regardless" part. Like a wildcard in a poker game, a wildcard in a filename can have any value.

The wildcard you see most often is * (an asterisk), but we'll start with something simpler: ? (a question mark). When it appears in a filename, the ? matches any single character. For example, letter? refers to any filename that begins with letter and has exactly one character after that. This would include letterA, letter1, as well as filenames with a nonprinting character as their last letter, such as letter^C.

The * wildcard matches any character or group of zero or more characters. For example, *.txt matches all files whose names end with .txt; c* matches all files whose names start with c; c*b* matches names starting with c and containing at least one b; and so on.

The * and ? wildcards are sufficient for 90 percent of the situations that you will find. However, there are some situations that they can't handle. For example, you may want to list files whose names end with .txt, mail, or let. There's no way to do this with a single *; it won't let you exclude the files you don't want. In this situation, use a separate * with each filename ending:

*.txt *mail *let

Sometimes you need to match a particular group of characters. For example, you may want to list all filenames that begin with digits or all filenames that begin with uppercase letters. Let's assume that you want to work with the files program.n, where n is a single-digit number. Use the filename:

program.[0123456789]

In other words, the wildcard [ character-list

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A multiuser system needs a way to let different users have different files with the same name. It also needs a way to keep files in logical groups. With thousands of system files and hundreds of files per user, it would be disastrous to have all of the files in one big heap. Even single-user operating systems have found it necessary to go beyond "flat" filesystem structures.

Almost every operating system solved this problem by implementing a tree-structured, or hierarchical, filesystem. Unix is no exception. A hierarchical filesystem is not much different from a set of filing cabinets at the office. Your set of cabinets consists of many individual cabinets. Each individual cabinet has several drawers; each drawer may have several partitions in it; each partition may have several hanging (Pendaflex) folders; and each hanging folder may have several files. You can specify an individual file by naming the filing cabinet, the drawer, the partition, the group of folders, and the individual folder. For example, you might say to someone: "Get me the `meeting of July 9' file from the Kaiser folder in the Medical Insurance Plans partition in the Benefits drawer of the Personnel file cabinet." This is backwards from the way you'd specify a filename, because it starts with the mfost specific part, but the idea is essentially the same.

You could give a complete path like this to any file in any of your cabinets, as shown in Figure 1-2. The concept of a "path" lets you distinguish your July 9 meeting with Kaiser from your July 9 interview with a job applicant or your July 9 policy-planning meeting. It also lets you keep related topics together: it's easy to browse through the "Medical Insurance" section of one drawer or to scan all your literature and notes about the Kaiser plan. The Unix filesystem works in exactly the same way (as do most other hierarchical filesystems). Rather than having a heap of assorted files, files are organized into directories. A directory is really nothing more than a special kind of file that lists a bunch of other files (see Section 10.2). A directory can contain any number of files (although for performance reasons, it's a good idea to keep the number of files in one directory relatively small — under 100, when you can). A directory can also contain other directories. Because a directory is nothing more than a special kind of file, directories also have names. At the top (the filesystem "tree" is really upside down) is a directory called the "root," which has the special name

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Microsoft Windows and the Mac OS have hierarchical filesystems (Section 1.14), much like those in Unix and other large systems. But there is an important difference. On many Windows and Mac systems, you start right at the "root" of the filesystem tree. In effect, you start with a blank slate and create subdirectories to organize your files.

A Unix system comes with an enormous filesystem tree already developed. When you log in, you start somewhere down in that tree, in a directory created for you by the system administrator (who may even be yourself, if you are administering your own system).

This directory — the one place in the filesystem that is your very own, to store your files (especially the shell setup files (Section 3.3) and rc files (Section 3.20) that you use to customize the rest of your environment) — is called your home directory.

Home directories were originally stored in a directory called /usr (and still are on some systems), but are now often stored in other directories, such as /home. Within the Linux Filesystem Hierarchy Standard (FHS), the home directory is always at /home, as configuration files are always in /etc and so on.

To change your current directory (Section 1.16) to your home, type cd with no pathname; the shell will assume you mean your home directory.

Within the Mac OS X environment, home is in the /Users/username directory by default.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Pathnames locate a file (or directory, or any other object) in the Unix filesystem. As you read this article, refer to Figure 1-4. It's a diagram of a (very) small part of a Unix filesystem.

Figure 1-4: Part of a Unix filesystem tree

Whenever you are using Unix, you have a current directory. By default, Unix looks for any mentioned files or directories within the current directory. That is, if you don't give an absolute pathname (Section 1.14) (starting from the root, / ), Unix tries to look up files relative to the current directory. When you first log in, your current directory is your home directory (Section 1.15), which the system administrator will assign to you. It typically has a name like /u/mike or /home/mike. You can change your current directory by giving the cd command, followed by the name of a new directory (for example,

cd

/usr/bin

). You can find out your current directory by giving the pwd ("print working directory") command.

If your current directory is /home/mike and you give the command cat textfile, you are asking Unix to locate the file textfile within the directory /home/mike. This is equivalent to the absolute path /home/mike/textfile. If you give the command cat notes/textfile, you are asking Unix to locate the file textfile within the directory notes, within the current directory /home/mike.

A number of abbreviations help you to form relative pathnames more conveniently. You can use the abbreviation . (dot) to refer to the current working directory. You can use .. (dot dot) to refer to the parent of the current working directory. For example, if your current directory is /home/mike, ./textfile is the same as textfile, which is the same as /home/mike/textfile. The relative path ../gina/textfile is the same as /home/gina/textfile; .. moves up one level from /home/mike (to /home) and then searches for the directory gina and the file textfile.

You can use either the abbreviation ~ (tilde) or the environment variables $HOME or $LOGDIR, to refer to your home directory. In most shells,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Under Unix, access to files is based on the concept of users and groups.

Every "user" on a system has a unique account with a unique login name and a unique UID (Section 24.3) (user ID number). It is possible, and sometimes convenient, to create accounts that are shared by groups of people. For example, in a transaction-processing application, all of the order-entry personnel might be assigned a common login name (as far as Unix is concerned, they only count as one user). In a research and development environment, certain administrative operations might be easier if members of a team shared the same account, in addition to having their own accounts. However, in most situations each person using the system has one and only one user ID, and vice versa.

Every user may be a member of one or more "groups." The user's entry in the master password file (/etc/passwd (Section 22.3)) defines his "primary group membership." The /etc/group (Section 49.6) file defines the groups that are available and can also assign other users to these groups as needed. For example, I am a member of three groups: staff, editors, and research. My primary group is staff; the group file says that I am also a member of the editors and research groups. We call editors and research my "secondary groups." The system administrator is responsible for maintaining the group and passwd files. You don't need to worry about them unless you're administering your own system.

Every file belongs to one user and one group. When a file is first created, its owner is the user who created it; its group is the user's primary group or the group of the directory in which it's created. For example, all files I create are owned by the user mikel and the group staff. As the file's owner, I am allowed to use the chgrp command to change the file's group. On filesystems that don't have quotas (Section 15.11), I can also use the chown command to change the file's owner. (To change ownership on systems with quotas, see Section 50.15.) For example, to change the file

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In general, a process (Section 24.1) is a program that's running: a shell, the ls command, the vi editor, and so on. In order to kill a process (Section 24.12), change its priority (Section 26.5), or manipulate it in any other way, you have to be the process' owner (i.e., the user who started it). In order to delete a job from a print queue (Section 45.1), you must be the user who started it.

As you might guess, there needs to be a way to circumvent all of this security. Someone has to be able to kill runaway programs, modify the system's files, and so on. Under Unix, a special user known as root (and commonly called the "superuser") is allowed to do anything.

To become the superuser, you can either log in as root or use the su (Section 49.9) command. In this book, though, we'll assume that you don't have the superuser password. Almost all of what we describe can be done without becoming superuser.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix differs from most operating systems in that it is file oriented. The designers of Unix decided that they could make the operating system much simpler if they treated everything as if it were a file. As far as Unix is concerned, disk drives, terminals, modems, network connections, etc. are all just files. Recent versions of Unix (such as Linux) have gone further: files can be pipes (FIFOs) (Section 43.11) and processes are files (Section 24.9). Like waves and particles in quantum physics, the boundary between files and the rest of the world can be extremely fine: whether you consider a disk a piece of hardware or a special kind of file depends primarily on your perspective and what you want to do with it.

Therefore, to understand Unix, you have to understand what files are. A file is nothing more than a stream of bytes — that is, an arbitrarily long string of bytes with no special structure. There are no special file structures and only a few special file types (for keeping track of disks and a few other purposes). The structure of any file is defined by the programs that use it, not by the Unix operating system. You may hear users talk about file headers and so on, but these are defined by the applications that use the files, not by the Unix filesystem itself.

Unix programs do abide by one convention, however. Text files use a single newline character (linefeed) between lines of text, rather than the carriage return-linefeed combination used in Microsoft Windows or the carriage returns used in the Macintosh. This difference may cause problems when you bring files from other operating systems over to Unix. Windows files will often be littered with carriage returns (Ctrl-M), which are necessary for that operating system but superfluous for Unix. These carriage returns will look ugly if you try to edit or print the file and may confuse some Unix programs. Mac text files will appear to be one long line with no breaks. Of course, you can use Unix utilities to convert Mac and Windows files for Unix.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Scripting languages and scripting applications differ from compiled languages and applications in that the application is interpreted as run rather than compiled into a machine-understandable format. You can use shell scripting for many of your scripting needs, but there are times when you'll want to use something more sophisticated. Though not directly a part of a Unix system, most Unix installations come with the tools you need for this more complex scripting — Perl (Chapter 41), Python (Chapter 42), and Tcl.

These three scripting languages seem so prevelant within the Unix world that I think of them as the Unix Scripting language triumvirate.

Perl is probably the granddaddy of scripting. Created by Larry Wall, this language is probably used more than any other for creating complex scripts to perform sophisticated functionality with Unix and other operating systems. The language is particularly noted for its ability to handle regular expressions, as well as working with files and other forms of I/O.

Python isn't as widespread as Perl, but its popularity is growing. One reason it's gaining popularity is that as a language, Python is more structured and a little more verbose than Perl, and therefore a little easier to read. In addition, according to its fans, Python has more object-oriented and data-manipulation features than the file-manipulation and regular-expression manipulation of Perl.

Tcl is particularly prevalent within Linux systems, though its use is widespread throughout all Unix systems. It's popular because it's simpler to learn than Perl and allows scripters to get up to speed more quickly than you can with Perl or Python. In addition, the language also has access to a very popular graphical user interface library called the Tk toolkit. You'll rarely hear about Tcl without the associated Tk.

—TOR and SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Generally speaking, a network lets two or more computers communicate and work together. Partly because of the open design of Unix, a lot of networking development has been done in this operating system. Just as there are different versions of Unix, there are different ways and programs to use networks from Unix.

There's an entire chapter devoted to Connectivity (Chapter 46), but for now, here's a quick review of the major networking components.

The Internet: The Internet is a worldwide network of computers. Internet users can transfer files, log into other computers, and use a wide range of programs and services.
WWW: The World Wide Web is a set of information servers on the Internet. The servers are linked into a hypertext web of documents, graphics, sound, and more. Point-and-click browser programs turn that hypertext into an easy-to-use Internet interface. (For many people, the Web is the Internet. But Unix lets you do much more.)
mail: A Unix facility that's been around for years, long before networking was common, is electronic mail. Users can send electronic memos, usually called email messages, between themselves. When you send email, your message waits for the other user to start his own mail program. System programs can send you mail to tell you about problems or give you information. You can send mail to programs, asking them for information. Worldwide mailing lists connect users into discussion groups.
ftp: The ftp program is one way to transfer files between your computer and another computer with TCP/IP, often over the Internet network, using the File Transfer Protocol (FTP).
UUCP: Unix-to-Unix Copy is a family of programs (uucp, uux, uulog, and others) for transferring files and email between computers. UUCP is usually used with modems over telephone lines and has been mostly superceded by Internet-type connections.
Usenet: Usenet isn't exactly a network. It's a collection of hundreds of thousands (millions?) of computers worldwide that exchange files called

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In 1988, an organization called the MIT (Massachusetts Institute of Technology) X Consortium was formed to promote and develop a vendor-neutral windowing system called the X Window System. (It was called "X" because it was a follow-on to a window system called "W" that was developed at Stanford University.) The organization eventually moved away from MIT and became known as the X Consortium. The XFree86 Project, Inc. is another major group developing X; they produce a freely redistributable version that's used on Linux and other Unix-like systems such as Darwin.

A window system is a way of dividing up the large screen of a workstation into multiple virtual terminals, or windows. Each window can interact with a separate application program — or a single application can have many windows. While the "big win" is to have applications with point-and-click mouse-driven user interfaces, one of the most common applications is still a simple terminal emulator (xterm (Section 5.9)). X thus allows a workstation to display multiple simultaneous terminal sessions — which makes many of the standard Unix multitasking features such as job control less important because programs can all be running in the foreground in separate windows. X also runs on many kinds of hardware, and it lets you run a program on a remote computer (across a network) while the program's windows are displayed on your local system. Because Unix systems also run on many kinds of hardware, this makes X a good match for Unix.

Unix boxes are, by default, characters-based systems. GUI systems are added to facilitate ease of use, as well as to provide access to a great number of sophisticated applications. The Mac OS X, though, is already a GUI, built on the BSD-based Unix environment, Darwin.

Though Darwin doesn't come with X Windows, you can download and install this, as well as X Windows-based GUIs, such as XDarwin (accessible at http://www.xdarwin.org) and OroborOSX (available at the Apple web site at http://www.apple.com).

—TOR and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Unix operating system was one of the first to include online documentation. It's not the best in the world — most users who haven't internalized the manual set curse it once a week — but it has proven surprisingly resilient. What's particularly interesting about Unix's online documentation is that, unlike other early help systems, it isn't an adjunct to another set of printed documentation that contains the "real" truth. The online manual is complete, authoritative, and usually more current than any printed documentation.

The basis for Unix's online documentation is the man command. Most simply, you use it as follows:

% man 

               topic

where topic is usually the name of some command; but it can also be the name of a system call, a library routine, an I/O device, or an administrative file (or file type). The output from man is usually sent to a pager like more, which allows you to page through the results.

There are several command-line options for the man command that can differ based on system. For instance, to look at a command within a specific section, on a System V machine use the -s "section" option, with the following format:

% man 

               section topic

% man -s 

               section topic

For example, if you want to read documentation about the /etc/passwd file (rather than the passwd command) on a System V machine, give the command:

% man -s 4 passwd

This is an easy way to distinguish between topics with the same name, but in different sections. For other Unix systems, such as FreeBSD, the option to search a section could be something different, such as -S.

Another useful command-line option is the -k option, which is equivalent to the apropos command. This option searches database files for matches of a given keyword, returning the results. This is particularly helpful in finding commands that contain a specific keyword if you're not quite sure what the command is.

Your system may have a configuration file for

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Unix operating system was one of the first to include online documentation. It's not the best in the world — most users who haven't internalized the manual set curse it once a week — but it has proven surprisingly resilient. What's particularly interesting about Unix's online documentation is that, unlike other early help systems, it isn't an adjunct to another set of printed documentation that contains the "real" truth. The online manual is complete, authoritative, and usually more current than any printed documentation.

The basis for Unix's online documentation is the man command. Most simply, you use it as follows:

% man 

               topic

where topic is usually the name of some command; but it can also be the name of a system call, a library routine, an I/O device, or an administrative file (or file type). The output from man is usually sent to a pager like more, which allows you to page through the results.

There are several command-line options for the man command that can differ based on system. For instance, to look at a command within a specific section, on a System V machine use the -s "section" option, with the following format:

% man 

               section topic

% man -s 

               section topic

For example, if you want to read documentation about the /etc/passwd file (rather than the passwd command) on a System V machine, give the command:

% man -s 4 passwd

This is an easy way to distinguish between topics with the same name, but in different sections. For other Unix systems, such as FreeBSD, the option to search a section could be something different, such as -S.

Another useful command-line option is the -k option, which is equivalent to the apropos command. This option searches database files for matches of a given keyword, returning the results. This is particularly helpful in finding commands that contain a specific keyword if you're not quite sure what the command is.

Your system may have a configuration file for

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

whatis is almost identical to apropos or the use of man -k (Section 2.1), but it requires a command name as an argument — rather than an arbitrary string. Why is this useful? Well, let's say you forget what cat (Section 12.2) does. On my system, apropos cat gives you several screenfuls of output. You may not want to read the entire manual page. But whatis cat gives you a nice one-line summary:

% whatis cat

cat (1V)      - concatenate and display

The whatis command is equivalent to man -f on most systems.

Before running whatis the first time on your system — particularly if you're running a standalone machine using FreeBSD, Linux, or Darwin — you'll want to run the makewhatis at /usr/libexec/makewhatis, which creates the whatis database by scanning the command names from the existing manpages.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The whereis command helps you to locate the executable file, source code, and manual pages for a program. I use it primarily as a sanity check; if I type cat useless.txt and get the message "cat: command not found," I immediately try

whereis

cat

. This gives me a lot of information about what went wrong: someone may have removed cat (Section 12.2) from the system, or my PATH (Section 35.6) environment variable may be set incorrectly, etc.

Output from whereis typically looks like this:

% whereis more

cat: /bin/cat /usr/share/man/man1/cat.1.gz

This says that the executable file is /bin/cat and the manual page is /usr/share/man/man1/cat.1.gz.

whereis has a few options worth mentioning:

-b: Only report the executable name
-m: Only report the location of the manual page
-s: Only search for source files
-u: Only issue a report if any of the requested information (executable, manual page, source) is missing

There are other options for modifying the list of directories through which whereis searches; if you need these, check your manual pages. In addition, the functionality and flags for whereis can differ between versions of Unix. For instance, much of the basic functionality of the command was removed in version 4.4 of FreeBSD as well as Darwin. Again, the manual pages will show you this information.

—ML and SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When the other techniques in this chapter don't find the information you want, you can try searching the online manual page (Section 2.1) files. You'll probably have to wade through a lot of stuff that you don't want to see, but this method can work when nothing else does. As an example, you remember that there's some command for chopping columns out of a file. You try man -k or apropos, but it only mentions colrm and pr, and those aren't what you want. You'll usually be able to narrow your search to one or two manual page sections (Section 2.1); here, you know that user commands are in section 1. So you go to the manual pages and do a case-insensitive search through all the files for "column" or "chop":

% cd /usr/man/man1

% egrep -i 'column|chop' *

awk.1:Add up first column, print sum and average:

colrm.1:colrm \- remove characters from specified columns within each line

  ...

cut.1:.IX  cut  ""  "\fIcut\fP \(em remove columns from file"

  ...

It's cut ! Notice that awk also handles columns, but apropos doesn't say so.

(I cheated on that example: there were other ways to find cut — using the synonym apropos field instead of apropos column, for instance. But this method does work in tougher cases.) To search the manual page files, you'll need to know where they're stored. There are lots of possibilities. If your system has an /etc/man.config file, it'll probably tell you. Otherwise, the directories /usr/man or /usr/share/man are good places to look. If the command is local, try /usr/local/man and maybe /opt (a big tree where find (Section 9.4) can help). If your system has fast find or locate (Section 9.18), try searching for man or */man*.

Your manpage files may be compressed (Section 15.6). In that case, use grep (Section 13.2) with the -Z option, grep -Z.

You'll probably find subdirectories with names like man1, man2, . . . and/or

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Each computer on a network needs a name. On many Unix versions, the uname -n command shows you this name. On some systems, the command hostname or uuname -l (two us, lowercase L) may be what you want. If you use more than one system, the hostname is great to use in a shell prompt — or any time you forget where you're logged in.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Your system may have several versions of a particular command — for instance, a BSD-compatible version in one directory and a System V-compatible version somewhere else (and you might have added a private version in your own bin directory (Section 7.4)). Which command you'll get depends on your PATH (Section 35.6) environment variable. It's often essential to know which version you're using. For example:

$ type sort

sort is /bin/sort

tells me exactly which version of the sort program I'm using. (On one system I've used, there were two sorts; I had also defined an alias for sort.) If I want to see all versions, bash supports a -all option:

$ type -all sort

sort is aliased to `TMPDIR=/var/tmp /bin/sort'

sort is /bin/sort

sort is /usr/5bin/sort

A similar command is whence .

But type and whence are built into shells and are also Unix-version dependent (not all Unix systems have them), so they won't work everywhere. The which command is usually external (Section 1.9), so it works everywhere — although, because it isn't built into the shells, it can't always find out about aliases defined in your current shell. For example:

% which sort

/usr/bin/sort

You'll find that which comes in handy in lots of other situations. I find that I'm always using which inside of backquotes to get a precise path. (whence and type may print extra text.) For example, when I was writing these articles, I started wondering whether or not man, apropos, and whatis were really the same executable. It's a simple question, but one I had never bothered to think about. There's one good way to find out:

% ls -li `which man` `which apropos` `which whatis`

102352 -rwxr-xr-x  3 root        24576 Feb  8  2001 /usr/ucb/apropos

102352 -rwxr-xr-x  3 root        24576 Feb  8  2001 /usr/ucb/man

102352 -rwxr-xr-x  3 root        24576 Feb  8  2001 /usr/ucb/whatis

What does this tell us? Well, within this system the three commands have the same file size, which means that they're likely to be identical; furthermore, each file has three links, meaning that each file has three names. The

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Each login session has its own tty (Section 24.6) — a Unix device file that handles input and output for your terminal, window, etc. Each tty has its own filename. If you're logged on more than once and other users want to write or talk (Section 1.21) to you, they need to know which tty to use. If you have processes running on several ttys, you can tell which process is where.

To do that, run the tty command at a shell prompt in the window:

% tty

/dev/tty07

You can tell other users to type write your-username tty07.

Most systems have different kinds of ttys: a few dialup terminals, some network ports for rlogin and telnet, etc. (Section 1.21). A system file like /etc/ttys lists which ttys are used for what. You can use this to make your login setup more automatic. For example, most network terminals on our computers have names like /dev/ttyp x or /dev/pts/ x, where x is a single digit or letter. I have a test in my .logout file (Section 4.17) that clears the screen on all ttys except network:

# Clear screen non-network ttys:

` ` Section 28.14

if ("`tty`" !~ /dev/ttyp?) then

    clear

endif

(Of course, you don't need to clear the terminal screen if you're using an xterm window that you close when you log out.)

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The who command lists the users logged on to the system now. Here's an example of the output on my system:

% who

naylor   ttyZ1   Nov  6 08:25

hal      ttyp0   Oct 20 16:04   (zebra.ora.com:0.)

pmui     ttyp1   Nov  4 17:21   (dud.ora.com:0.0)

jpeek    ttyp2   Nov  5 23:08   (jpeek.com)

hal      ttyp3   Oct 28 15:43   (zebra.ora.com:0.)

    ...

Each line shows a different terminal or window. The columns show the username logged on, the tty (Section 2.7) number, the login time, and, if the user is coming in via a network (Section 1.21), you'll see their location (in parentheses). The user hal is logged on twice, for instance.

It's handy to search the output of who with grep (Section 13.1) — especially on systems with a lot of users. For example:

% who | grep "^hal "             

               ...where is hal logged on?

% who | grep "Nov  6"            

               ...who logged on today?

-v Section 13.3

% who | grep -v "Nov  6"         

               ...who logged on before today?

    ...

The GNU who, on the CD-ROM [see http://examples.oreilly.com/upt3], has more features than some other versions.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

An information system gaining popularity on the more lightweight Unix-based systems is info. It's particularly relevant for finding information within Linux and FreeBSD.

Unlike man — which displays all information on a topic at once, usually routed through some form of paging system such as cat — info is based on a hypertext like linkage between topic components. You connect to each of the subtopics using character-based commands and typing part or all of the subtopic title — at least enough to distinguish one subtopic from another.

To use info, you type the command info followed by the Unix command about which you're trying to find information. For instance, to find out more about info itself, you would use the following command line:

info info

This will return the main info introduction page and a menu of subtopics such as:

Getting Started

Advanced Info

Creating an Info File

To access the subtopic, you type the letter m for menu, and then in the prompt that opens at the bottom of the screen, type enough of the letters to distinguish the subtopic menu item from any other. You don't have to complete the command: you can just type enough of the letters followed by a TAB to fill in the rest. Once the subtopic menu item has been filled in, hitting ENTER sends you to the information.

To learn more about using info, you can type the letter h when you're in info and no command line buffer is showing. This brings up basic information about the info command, including the commands you use within info to use the application. These letters are summarized in Table 2-1.

Table 2-1: info commands
Command	Action
h	To get help on using info
m	To access a subtopic menu item

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you first log in to a Unix system, the login program performs various security measures. These vary slightly from Unix variant to Unix variant, but they are largely the same.

First, login checks to see if you are not the root user and whether the file /etc/nologin exists (it is created by the shutdown command to indicate that the system is being brought down for maintenance). If both of these conditions are true, the login program prints the contents of that file to the terminal, and the login fails. If you are the root user, however, you will be allowed to log in.

Second, login checks to see if there are any special conditions on your login attempt (which may be defined in /etc/usertty or /etc/securetty), such as on which tty you're coming in. Linux systems do this, for example. Some systems (such as Darwin and other BSD-based systems) also check /etc/fbtab and may restrict your access to any devices listed in that file. These systems may also log failed login attempts to a file, such as /var/log/failedlogin, if it exists.

login may also record the time of login in the file /var/log/lastlog, make an entry in the file /var/run/utmp , showing that you are successfully logged in (it is removed once you log out), and append the utmp entry to the file /var/log/wtmp , showing that you logged in. This wtmp record will be updated on logout, showing the duration of your login session.

If the file .hushlogin exists in the user's home directory, the login will be quiet; otherwise, the following sequence of events will occur. If the system has any special copyright information to display, it will be printed to the terminal, followed by the message of the day (usually stored in /etc/motd ), and the user's last login time and system of origin (from the wtmp file, as discussed in the previous paragraph). If you want your login to be quiet, simply touch ~/.hushlogin. If you want it to be noisy, remove the file.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you first log in to a Unix system, the login program performs various security measures. These vary slightly from Unix variant to Unix variant, but they are largely the same.

First, login checks to see if you are not the root user and whether the file /etc/nologin exists (it is created by the shutdown command to indicate that the system is being brought down for maintenance). If both of these conditions are true, the login program prints the contents of that file to the terminal, and the login fails. If you are the root user, however, you will be allowed to log in.

Second, login checks to see if there are any special conditions on your login attempt (which may be defined in /etc/usertty or /etc/securetty), such as on which tty you're coming in. Linux systems do this, for example. Some systems (such as Darwin and other BSD-based systems) also check /etc/fbtab and may restrict your access to any devices listed in that file. These systems may also log failed login attempts to a file, such as /var/log/failedlogin, if it exists.

login may also record the time of login in the file /var/log/lastlog, make an entry in the file /var/run/utmp , showing that you are successfully logged in (it is removed once you log out), and append the utmp entry to the file /var/log/wtmp , showing that you logged in. This wtmp record will be updated on logout, showing the duration of your login session.

If the file .hushlogin exists in the user's home directory, the login will be quiet; otherwise, the following sequence of events will occur. If the system has any special copyright information to display, it will be printed to the terminal, followed by the message of the day (usually stored in /etc/motd ), and the user's last login time and system of origin (from the wtmp file, as discussed in the previous paragraph). If you want your login to be quiet, simply touch ~/.hushlogin. If you want it to be noisy, remove the file.

Finally, if all other checks have passed and restrictions have been performed,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Throughout the book, we will refer to terminals, terminal emulators, and other software that allows you, the end user, to interact with the computer via some character-driven screen. In the old days, most terminals were separate hardware, but nowadays they're usually software. Mac OS X is no exception: its Terminal application, found in the Utilities folder of your Applications folder, is a terminal emulator.

You can launch Terminal by double-clicking on the icon in the Finder, or if you have the Terminal icon in your Dock, by single-clicking on that icon.

Once launched, Terminal may be configured as most Mac applications can: by setting preferences in the Preferences dialog and choosing a font family and size from the Font menu.

One big difference between Terminal and other, X-specific applications is that instead of running individual instances of xterm, you run one instance of Terminal and may have multiple windows, known as "shells," which may have saved settings (such as color, size, font choice, and various other settings). You can't run a shell in Mac OS X without running Terminal.

— SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

To understand setup files, you need to understand that a shell can act like a login shell or a nonlogin shell (Section 3.4). There are different setup files for nonlogin and login shells.

When you log in to a Unix system — but not under a window system — the login program starts a shell for you. The login program sets a special flag (Section 3.19) to tell a shell that it's a login shell. If the shell doesn't have that flag set, it won't act like a login shell. Opening a new window in a window system may or may not set the "login shell" flag — that depends on the configuration. (For example, the command xterm -ls starts a login shell in an xterm window (Section 24.20); xterm +ls starts a nonlogin shell.) When you connect to a system with programs like ftp and scp, that usually starts a nonlogin shell. And a subshell (Section 24.4) is never a login shell (unless you set a command-line option to force a login shell, like bash -l).

How can you tell whether your shell is a login shell? The answer is "it depends." When you first log in to a system, you want a login shell that sets things like the terminal type (Section 5.2, Section 5.3). Other shells on the same terminal should be nonlogin shells — to avoid redoing those one-time-only setup commands. Different shells have their own methods for handling first-time shell invocations versus later invocations, and that's what the rest of this article is about.

Parenthesis operators (Section 43.7) don't read any setup file. Instead, they start another instance of your current shell. Parentheses are called "subshell operators," but the subshell they start doesn't print a prompt and usually has a short lifetime.

Next, let's look at the setup files — login and nonlogin — for the major shells. I recommend that you read about all of them. Then experiment with your shell's setup files until you get things working the way you want them.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Each Unix shell (sh, csh, etc.) can be in interactive mode or noninteractive mode. A shell also can act as a login shell or a nonlogin shell. A shell is a shell is a shell — e.g., a login bash shell is the same program (like /bin/bash) as a nonlogin bash shell. The difference is in the way that the shell acts: which setup files it reads, whether it sets a shell prompt, and so on.

When you first log in to a Unix system from a terminal, the system normally starts a login shell. (Section 3.4) A login shell is typcally the top-level shell in the "tree" of processes that starts with the init (Section 24.2) process. Many characteristics of processes are passed from parent to child process down this "tree" — especially environment variables (Section 35.3), such as the search path (Section 35.6). The changes you make in a login shell will affect all the other processes that the top-level shell starts — including any subshells (Section 24.4).

So, a login shell is where you do general setup that's done only the first time you log in — initialize your terminal, set environment variables, and so on. A shell "knows" (Section 3.19) when it's a login shell — and, if it is, the shell reads special setup files (Section 3.3) for login shells. For instance, login C shells read your .login file, and Bourne-type login shells read .profile. Bash may also read /etc/profile, and ~/.bash_profile or ~/.bash_login or ~/.profile, depending on whether those files exist and whether the -noprofile option has been passed (which would disable reading of any startup files).

Nonlogin shells are either subshells (started from the login shell), shells started by your window system (Section 24.20), or "disconnected" shells started by at (Section 25.5), rsh (Section 1.21), etc. These shells don't read .login or .profile. In addition, bash allows a nonlogin shell to read ~/.bashrc or not, depending on whether the -norc or -rcfile

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Setup files for login shells ( Section 3.4) — such as .login and .profile — typically do at least the following:

Set the search path ( Section 27.6) if the system default path isn't what you want.
Set the terminal type ( Section 5.3) and make various terminal settings (Section 5.7, Section 5.8) if the system might not know your terminal (if you log in from various terminals over a dialup line or from a terminal emulator on a desktop machine, for instance).
Set environment variables ( Section 35.3) that might be needed by programs or scripts that you typically run.
Run one or more commands that you want to run whenever you log in. For example, if your system login program doesn't show the message of the day, your setup file can. Many people also like to print an amusing or instructive fortune. You also might want to run who (Section 2.8) or uptime (Section 26.4) or w (a combination of the other two, but not found on all systems) for information about the system.

In the C shell, the .cshrc file is used to establish settings that will apply to every instance of the C shell, not just login shells. For example, you typically want aliases (Section 28.2) available in every interactive shell you run — but these aren't passed through the environment, so a setup file has to do the job. You may wish to put all of your aliases into another file, such as .aliases, or qualify the name with the shell's name, such as .csh.aliases, to allow for different alias formats between shells, and then you can use the source command to read in that file on startup from .cshrc.

Even novices can write simple setup files. The trick is to make these setup scripts really work for you. Here are some of the things you might want to try:

Creating a custom prompt.
Coordinating custom setup files on different machines (Section 3.18).
Making different terminal settings depending on which terminal you're using (Section 3.10 and others).
Seeing the message of the day only when it changes.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The shell is your interface to Unix. If you make a bad mistake when you change your setup file (Section 3.3) or your password, it can be tough to log in and fix things.

Before you change your setup, it's a good idea to start a login session to the same account from somewhere else. Use that session for making your changes. Log in again elsewhere to test your changes.

Don't have a terminal with multiple windows or another terminal close to your desk? You can get the same result by using rlogin or telnet (Section 1.21) to log in to your host again from the same terminal. What I mean is:

somehost% vi .cshrc

               ...Make edits to the file...

somehost% rlogin localhost

               ...Logs you in to your same account...

               An error message

somehost% logout

Connection closed.

somehost% vi .cshrc

               ...Edit to fix mistake...

If you don't have rlogin or telnet, the command su - username, where username is your username, will do almost the same thing. Or, if you're testing your login shell configuration, login will do as well.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One common mistake in shell setup files (Section 3.3) is lines like these:

$$ Section 27.17, `...` Section 28.14

source .aliases



echo "Shell PID $$ started at `date`" >> login.log

What's wrong with those lines? Both use relative pathnames (Section 1.16) for the files (.aliases, login.log), assuming the files are in the home directory. Those lines won't work when you start a subshell (Section 24.4) from somewhere besides your home directory because your setup files for nonlogin shells (like .cshrc) are read whenever a shell starts. If you ever use the source or . commands (Section 35.29) to read the setup files from outside your home directory, you'll have the same problem.

Use absolute pathnames instead. As Section 31.11 explains, the pathname of your home directory is in the tilde (~) operator or the $HOME or $LOGDIR environment variable:

source ~/.aliases

echo "Shell PID $$ started at `date`" >> ~/login.log

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The C shell reads its .cshrc, .login, and .logout setup files at particular times (Section 3.3). Only "login" C shells (Section 3.4) will read the .login and .logout files. Back when csh was designed, this restriction worked fine. The shell that started as you logged in was flagged as a login shell, and it read all three files. You started other shells (shell escapes, shell scripts, etc.) from that login shell, and they would read only .cshrc. The same can be said of other shell variants, such as tcsh, though they may have multiple startup files — the problem of distinguishing between login and nonlogin shell startup is the same.

Now, Unix has interactive shells started by window systems (like xterm (Section 24.20)), remote shells (like rsh (Section 1.21) or ssh), and other shells that might need some things set from the .login or .logout files. Depending on how these shells are invoked, these might not be login shells — so they might read only .cshrc (or .tcshrc, etc.). How can you handle that? Putting all your setup commands in .cshrc isn't a good idea because all subshells (Section 24.4) read it . . . you definitely don't want to run terminal-setting commands like tset (Section 5.3) during shell escapes!

Most other shells have the same problem. Some, like zsh and bash, have several setup files that are read at different times — and probably can be set up to do what you want. For other shells, though, you'll probably need to do some tweaking.

To handle problems at login time, put almost all of your setup commands in a file that's read by all instances of your shell, login or nonlogin. (In the C shell, use .cshrc instead of .login.) After the "login-only" commands have been read from the setup file, set the ENV_SET environment variable (Section 35.3) as a flag. (There's nothing special about this name. You can pick any name you want.) You can then use this variable to test whether the login-only commands have already been run and skip running them again in nonlogin shells.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Lots of users add an if (! $?prompt) exit test to their .cshrc files. It's gotten so common that some vendors add a workaround to defeat the test. For instance, some versions of the which command (Section 2.6) set the prompt variable so that it can see your aliases "hidden" inside the $?prompt test. I've also seen a version of at that starts an interactive shell to run jobs.

If you've buried commands after if (! $?prompt) that should only be run on interactive shells or at login time, then you may have trouble.

There are workarounds. What you'll need depends on the problem you're trying to work around.

The version of which on the CD-ROM [see http://examples.oreilly.com/upt3] works without reading your .cshrc file, so there's no problem there.
Here's a way to stop the standard which from reading parts of your .cshrc that you don't want it to read. The first time you log in, this scheme sets a CSHRC_READ environment variable (Section 35.3). The variable will be copied into all subshells ( Section 24.4) (like the one that which starts). In subshells, the test if ($?CSHRC_READ) will branch to the end of your .cshrc file:
```
if (! $?prompt) goto cshrc_end



# COMMANDS BELOW HERE ARE READ ONLY BY INTERACTIVE SHELLS:

alias foo bar

   ...



if ($?CSHRC_READ) goto cshrc_end





# COMMANDS BELOW HERE ARE READ ONLY AT LOGIN TIME:

setenv CSHRC_READ yes

   ...



cshrc_end:
```
If you have a buggy version of at (Section 25.5) that runs jobs from interactive shells, make your own frontend to at (Section 29.1) that sets an environment variable named AT temporarily before it submits the at job. Add a test to your .cshrc that quits if AT is set:
( ) Section 43.7, \at Section 29.8

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you work at several kinds of terminals or terminal emulators, terminal setup can be tough. For instance, my X terminal sends a backspace character when I push the upper-right key, but the same key on another terminal sends a delete character — I want stty erase (Section 5.8) to set the correct erase character automatically. Maybe you want a full set of calendar programs started when you log in to the terminal at your desk, but not when you make a quick login from somewhere else.

The next seven articles have ideas for changing your login sequence automatically. Some examples are for the C shell and use that shell's switch and if. Examples for Bourne-type shells use case (Section 35.10) and if (Section 35.13). If you use the other type of shell, the idea still applies; just swap the syntax.

If you use several kinds of terminals or terminal emulators, try testing the TERM environment variable (Section 3.11). Testing other environment variables (Section 3.14) can identify the frontend system (like a window system) you're using.
Test the output of who am i (Section 3.12) to find out about the remote system from which you've logged in.
If you log into different kinds of ports — network, hardwired, and so on — search for the port type (Section 3.15) in a table like /etc/ttys (in BSD derivatives) or /etc/inittab (in some other variants). Testing the port name (Section 3.13) may also work.
In the X Window System, you can test the window size (Section 3.16) and make various settings based on that. Naming windows (Section 3.17) lets you identify a particular window by reading its environment.
You can also handle some of these cases using the venerable but obscure tset (Section 5.3) program to select and initialize the correct terminal type. Another program that sets the terminal type is qterm (Section 5.4).

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you use several different kinds of terminals (or, as is far more common these days, terminal emulators) and your TERM environment variable is set differently on each terminal, you can add a test like this to your C shell .login file:

switch ($TERM)

case vt100:

   ...do commands for vt100

   breaksw

case xxx:

   ...do commands for xxx

   breaksw

default:

   ...do commands for other terminals

   breaksw

endsw

If you have a Bourne-type shell, use a case statement (Section 35.10) in your .profile instead:

case "$TERM" in

    vt100)

        ...do commands for vt100

        ;;

    xterm)

        ...do commands for xterm

        ;;

    *)

        ...do commands for other terminals

        ;;

esac

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you log in from other hosts (Section 1.21) or from hosts running the X Window System (Section 24.20), the who am i command will probably show a hostname and/or window information in parentheses:

schampeo@fugazi:1002 $ who am i

schampeo ttyp7    Jun 19 03:28 (fugazi:0.0)

(Long hostnames may be truncated. Also, note that some versions of who am i prepend the name of the local host to the username and don't include the remote hostname at all in their output. Check yours before you write this test.) The information in parentheses can help you configure your terminal based on where you've logged in from and/or which display you're using. To test it, add commands such as the following to your .profile file. (In C-type shells, use a switch statement in .login instead.)

case Section 35.10

case "`who am i | sed -n 's/.*(\(.*\))/\1/p'`" in

$..$ \1 Section 34.11

*0.0)   ...do commands for X display 0 ;;

mac2*)  ...do commands for the host mac2.foo.com ;;

"")     ...no output (probably not a remote login) ;;

*)      ...do commands for other situations ;;

esac

That uses sed (Section 34.1) to give the text between the parentheses for that remote host to the case. This *0.0 case matches lines ending with 0.0; the mac2 case matches lines that start with mac2; an empty string means sed probably didn't find any parentheses; and the * case catches everything else.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you know that certain port (tty) numbers are used for certain kinds of logins, you can test that and change your terminal setup based on the tty you're currently using. For example, some systems use ttyp0, ttyq1, etc. as network ports for rlogin and ssh (Section 1.21), while others use pty0, etc. This Bourne-type case statement branches on the port name:

tty Section 2.7

case "`tty`" in

/dev/tty[pqrs]?)

   # rlogin, telnet:

   ...

/dev/tty02)

   # terminal on my desk:

   ...

"not a tty") ;;  ...not a terminal login session; do nothing

esac

In C-type shells, try a switch or if statement instead.

On Linux, you may need to look for patterns to match /dev/pts/0, /dev/pts/1, etc.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Certain systems set certain environment variables. For example, the X Window System sets a DISPLAY environment variable (Section 35.5). If you've logged in from a remote system using ssh (Section 1.21), look for variables like SSH_CLIENT and SSH_TTY or SSH_AUTH_SOCK on the system you log in to. (If you aren't sure about your system, use the env or printenv command (Section 35.3) to look for changes in your environment at different systems.)

Your shell setup file (Section 3.3) makes decisions based on the environment variables that have been set. Here are examples for both C-type and Bourne-type shells:

[ ] Section 35.26

if ($?DISPLAY) then                 if [ -n "$DISPLAY" ]; then

   # on X window system                # on X window system

   ...                                 ...

else if ($?XDARWIN_VERSION) then    elif [ -n "$XDARWIN_VERSION" ]; then

   # on MacOS X system                 # on MacOS X system

   ...                                 ...

else                                else

   ...                                 ...

endif                               fi

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Your system may have an /etc/ttytab or /etc/ttys file that lists the type of each terminal port (tty (Section 24.6)). Here are lines from /etc/ttys on a NetBSD system I use:

console "/usr/libexec/getty std.9600"    vt100      on  local

tty00   "/usr/libexec/getty std.9600"    dialup     off local

tty01   "/usr/libexec/getty std.9600"    plugboard  off local

   ...

ttyp0   none                             network    off

   ...

For example, port ttyp0 is network, the type used by xterm (Section 24.20), telnet (Section 1.21), etc.

To change your account configuration based on the tty port type, match the first column of that file to the output of the tty (Section 2.7) command, which shows your current tty pathname. The output of tty starts with /dev or /dev/pts. So, to match your current tty to the file, you need to strip the name to its tail. For example, in bash and ksh, these three lines would put the terminal port type (vt100, plugboard, etc.) into the ttykind shell variable:

tty=`tty`

ttytail=${tty#/dev/}

awk Section 20.10

ttykind=`awk '$1 == "'$ttytail'" {print $3}' /etc/ttys`

Then you can test the value with case (Section 35.10) or if (Section 35.13). In C shells, you can set ttytail by using the :t string modifier (Section 28.5) and test its value with switch or if.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I use several terminal windows of different sizes. I don't stretch the windows after I open them; instead, I set the size as I start each xterm . Here's an excerpt from my X setup file (Section 3.20) that opens the windows:

-e Section 5.22

xterm -title SETI -geometry 80x9+768+1 -e setiathome -verbose -nice 10 &

xterm -title "work xterm" -geometry 80x74+329-81 &

The first window has 9 rows (80x9) and the second has 74 rows (80x74). I'd like the less (Section 12.3) pager to use different jump-target lines in larger windows. If the window has more than 24 lines, I want less to use its option -j3 to show search-matches on the third line of the window instead of the first.

On many systems, the command stty size gives the number of rows and columns in the current window, like this:

$ stty size

74 80

Your system might need the command stty -a instead — or it could have environment variables named LINES and COLUMNS. We'll use stty size in the following Bourne shell setup file. The set (Section 35.25) command puts the number of rows into the $2 shell parameter. (Using set this way is portable to all shells, but it's a clumsy way to split stty's output into words. If you have a newer shell with array support, it'll be easier.) Then a series of if (Section 35.13)/then (Section 35.26) tests handle different window sizes:

LESS=emqc; export LESS

# Put number of rows into $2, configure session based on that:

set x `stty size`

if [ -z "$2" -o "$2" -lt 1 ]

then echo ".profile: bogus number of rows ($2) in window!?" 1>&2

elif [ "$2" -gt 24 ]

then LESS=j3$LESS

   ...

fi

Additionally, you may be able to run resize on machines with the X Window System installed; it may output something like this:

schampeo@fugazi:1046 $ resize

COLUMNS=80;

LINES=37;

export COLUMNS LINES;

You may then capture the output and read it for the current setting or simply check the COLUMNS or LINES environment variables.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I use several xterm windows. Here's an excerpt from my X setup file (Section 3.20):

WINNAME=console xterm -C -title Console -geometry 80x9+0+0 &

WINNAME=work xterm -title "work xterm" -geometry 80x74+329-81 &

The WINNAME= name sets an environment variable named WINNAME for the particular command line it's on. This is passed through the environment, through the xterm process, to the shell running inside the window. So the shell's setup file can test for this variable — and, by knowing the window name stored in that variable, do specific setup for just that window. For example, in tcsh :

-f Section 11.10, { } Section 28.4

if ($?WINNAME) then

    switch ($WINNAME)

    case console:

        # Watch logs:

        tail -f /var/log/{messages,maillog,secure} ~/tmp/startx.log &

        breaksw

    case work:

        /usr/games/fortune

        fetchmail

        breaksw

    endsw

endif

On the console terminal, this .tcshrc file starts a job in the background (Section 23.2) to watch log files. On the work xterm, I get a fortune and grab email from the POP server.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I work with different types of machines every day. It is often necessary to set things up differently for, say, a Linux box than a SPARCstation or a MacOS X box. Going beyond that, you may want to set things up differently on a per-host basis.

I have this test in my .cshrc file:

setenv Section 35.3

setenv HOST "`uname -n`"

~ Section 31.11

if (-e ~/lib/cshrc.hosts/cshrc.$HOST) then

   source ~/lib/cshrc.hosts/cshrc.$HOST

endif

So, if I log in to a machine named (Section 2.5) bosco, and I have a file called ~/lib/cshrc.hosts/cshrc.bosco, I can source (Section 35.29) it to customize my environment for that one machine. These are examples of things you would put in a .cshrc.$HOST file:

Search path (Section 27.6): Some machines have /usr/local/bin, and some have /opt. The same goes for cdpath (Section 31.5).
Terminal settings (Section 5.8): I always like to reach for the upper-right part of a keyboard to erase characters. Sometimes this is the location for the BACKSPACE key, and sometimes it is the DELETE key. I set things up so that I can consistently get "erase" behavior from whatever key is there.
Other shell variables (Section 35.9) and environment variables (Section 35.3): These may be different. You may run a package on a certain machine that relies on a few environment variables. No need to always set them and use up a little bit of memory if you only use them in one place!

In general, this idea allows you to group together whatever exceptions you want for a machine, rather than having to write a series of switch or if statements throughout your .cshrc and .login files. The principle carries over directly to the newer shells as well.

—DS and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you log in to most Unix systems, your shell is a login shell. When a shell is a login shell, it acts differently (Section 3.4).

Sometimes, when you're testing an account or using a window system, you want to start a login shell without logging in. Unix shells act like login shells when they are executed with a name that starts with a dash (-). This is easy to do if you're using a system call in the exec(3) family. These system calls let a C-language programmer give both the filename of an executable file, like sh or /bin/sh, as well as the name that should be used to identify the process (in a ps (Section 24.5) listing, for example), like -sh.

If you're currently using zsh , you can invoke another shell this way by typing a dash and a space before the shell's name:

zsh% - csh

               ...C shell starts, acting like a login shell...

%

C programmers can write a little program that runs the actual shell but tells the shell that its name starts with a dash. This is how the Unix login process does it:

run_login_csh( )

{

    execl("/bin/csh", "-csh", 0);

}

A more general solution is to make a link (Section 10.4) to the shell and give the link a filename starting with a dash. If your own bin subdirectory is on the same filesystem as /bin (or wherever the executable shell file is), you can use a hard link. Otherwise, make a symbolic link, as shown here:

bin Section 7.4, ./- Section 14.13

$ cd $HOME/bin

$ ln -s /bin/csh ./-csh

Then you can execute your new shell by typing its name:

$ -csh

               ...normal C shell login process...

% ...run whatever commands you want...

% logout

$ ...back to original shell

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One way to set defaults for your applications is with environment variables (Section 35.3) that the applications might read. This can get messy, though, if your environment has tens or hundreds of variables in it. A lot of applications have a different way to choose defaults: setup files, similar to shell setup files (Section 3.3). Most of these filenames end with rc, so they're often called RC files. Today's more-complex applications also use their own setup subdirectories. Almost all of these files and directories are hidden (Section 8.9) in your home directory; you'll need ls -A to see them.

This article describes some of the most common setup files. For a more complete list, check your application's manpage:

.emacs

For the Emacs editor. See Section 19.3.

.exrc

For the vi (actually, ex) editor. See Section 17.5.

.inputrc

For the GNU Readline library and applications that use it, such as the bash shell.

.mailrc

For the mail (Section 1.21) program and others like it. This can be handy if you use mail from the command line to send quick messages. For example:

# If I send mail to "bookquestions", send it to myself too:

alias bookquestions bookquestions@oreilly.com, jerry

# When I send a message, prompt me for "cc:" addresses:

set askcc

.mh_profile

For the MH email system.

.netrc

A listing of hostnames, accounts — and possibly passwords — used for connecting to remote hosts with ftp and some other programs. Should have file access mode (Section 50.2) 600 or 400 for security, but this may not be enough protection for passwords! Best used for Anonymous ftp.

.newsrc

For news readers (Section 1.21). (Some newer news readers have more complex files.) A list of newsgroups in the order you want to see them. For example:

comp.security.announce: 1-118

news.announce.important: 1

comp.org.usenix: 1-1745

comp.sys.palmtops! 1-55069,55071

   ...

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We strongly suggest that you write a manual page for each command that you place in your bin directory. Unix manual pages typically have the following format, which we suggest you follow:

NAME

     The program's name; one line summary of what it does.



SYNOPSIS

     How to invoke the program, including all arguments and

     command-line options. (Optional arguments are placed in

     square brackets.)



DESCRIPTION

     A description of what the program does—as long as

     is necessary.



OPTIONS

     An explanation of each option.



EXAMPLES

     One or more examples of how to use the program.



ENVIRONMENT

     Any environment variables that control the program's behavior.



FILES

     Files the program internals will read or write. May include

     temporary files; doesn't include files on the command line.



BUGS

     Any known bugs. The standard manual pages don't take

     bug recording seriously, but this can be very helpful.



AUTHOR

     Who wrote the program.

To see how a "real" manual page looks, type man ls.

Feel free to add any other sections that you think are necessary. You can use the nroff -man macros (Section 3.22) if you want a nicely formatted manual page. However, nroff is fairly complicated and, for this purpose, not really necessary. Just create a text file that looks like the one we showed previously. If you are using a BSD system and want your manual pages formatted with nroff, look at any of the files in /usr/man/man1, and copy it.

If you insist on formatting your manual page properly, using the troff or groff "man" macros, you can use nroff to preview the file.

The man (Section 2.1) command is essentially the same as this:

-s Section 11.7

% nroff -e -man 

               filename

                | more -s

Go to http://examples.oreilly.com/upt3 for more information on: gnroffawf

You can safely omit the -e option to nroff and the -s option to more, or even substitute in your favorite pager, such as less. And remember that

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're not satisfied with the simple manual pages we discussed in Section 3.21, here's how to go all the way and create a "real" manual page. As we said, the best way to create a manual page is to copy one that already exists. So here's a sample for you to copy. Rather than discuss it blow by blow, I'll include lots of comments (these start with .\" or \").

.\" Title: Program name, manual section, and date

1 Section 2.1

.TH GRIND 1

.\" Section heading: NAME, followed by command name and one line summary

.\" It's important to copy this exactly; the "whatis" database (used

.\" for apropos) looks for the summary line.

.SH NAME

grind \- create output from input

.\" Section heading: SYNOPSIS, followed by syntax summary

.SH SYNOPSIS

.B grind              \" .B: bold font; use it for the command name.

[ -b ] [ -c ] [ -d ]  \" Put optional arguments in square brackets.

[ input [ output ]]   \" Arguments can be spread across several lines.

.br                   \" End the synopsis with an explicit line break (.br)

.\" A new section: DESCRIPTION, followed by what the command does

.SH DESCRIPTION

.I Grind       \" .I:  Italic font for the word "Grind"

performs lots of computations. Input to

.IR grind ,    \" .IR: One word italic, next word roman, no space between.

is taken from the file

.IR input ,

and output is sent to the file

.IR output ,

which default to standard input and standard output if not specified.

.\" Another section: now we're going to discuss the -b, -c, and -d options

.SH OPTIONS

.\" The .TP macro precedes each option

.TP

.B \-b  \" print the -b option in bold.



Print output in binary.

.TP

.B \-c  \" \- requests a minus sign, which is preferable to a hyphen (-)

Eliminate ASCII characters from input before processing.

.TP

.B \-d

Cause daemons to overrun your computer.

.\" OK, we're done with the description and the options; now mention

.\" other requirements (like environment and files needed) as well as

.\" what can go wrong. You can add any other sections you want.

.SH FILES

.PD 0

.TP 20

.B /dev/null

data file

.TP

.B /tmp/grind*

temporary file (typically 314.159 Gigabytes)

.PD

.SH BUGS

In order to optimize computational speed, this program always produces

the same result, independent of the input.

.\" Use .LP between paragraphs

.LP

If the moon is full,

.I grind

may destroy your input file. To say nothing of your sex life.

.\" Good manual pages end by stating who wrote the program.

.SH AUTHOR

I wouldn't admit to this hack if my life depended on it.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The prompt displayed by your shell is contained in a shell variable (Section 35.9) called prompt in C-type shells and PS1 in Bourne-type shells. As such, it can be set like any other shell variable.

There are two or three ways to set a prompt. One is a static prompt (Section 4.2) that doesn't change during your login session (as you change directories, as the time of day changes, etc.). Some shells let you set a dynamic prompt (Section 4.3) string that is interpreted by the shell before each prompt is printed. Even on shells that don't interpret prompt strings dynamically, you can simulate a dynamic prompt (Section 4.4) by changing the prompt string automatically.

Depending on your shell's capabilties, you can use or combine those three techniques — and those found in the rest of this chapter — to do a lot. But, of course, you don't want to type that prompt-setting command every time you log in. So after you've perfected your prompt on the command line, store it in the correct shell setup file (Section 3.3): use the file that's read by interactive shells or add an interactive shell test to your setup file. (Setting the prompt in noninteractive shells is pointless — and it can even cause problems (Section 4.5).)

—JP, TOR, and SJC

As Section 4.1 explains, the simplest prompts — which I call static prompts — are prompts whose value are set once. The prompt doesn't change (until you reset the prompt variable, of course).

The default bash prompt is a good example of a static prompt. It's "bash$ " (with a space at the end, to make the command you type stand out from the rest of the prompt). You could set that prompt with the simple command:

PS1='bash$ '.

Notice the single quotes (Section 11.3) around the value; this is a good idea unless you want special characters in the prompt value to be interpreted before it's set. You can try it now: type that command on a command line, just as you would to set any other shell variable. Experiment a bit. The same prompt works on

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The prompt displayed by your shell is contained in a shell variable (Section 35.9) called prompt in C-type shells and PS1 in Bourne-type shells. As such, it can be set like any other shell variable.

There are two or three ways to set a prompt. One is a static prompt (Section 4.2) that doesn't change during your login session (as you change directories, as the time of day changes, etc.). Some shells let you set a dynamic prompt (Section 4.3) string that is interpreted by the shell before each prompt is printed. Even on shells that don't interpret prompt strings dynamically, you can simulate a dynamic prompt (Section 4.4) by changing the prompt string automatically.

Depending on your shell's capabilties, you can use or combine those three techniques — and those found in the rest of this chapter — to do a lot. But, of course, you don't want to type that prompt-setting command every time you log in. So after you've perfected your prompt on the command line, store it in the correct shell setup file (Section 3.3): use the file that's read by interactive shells or add an interactive shell test to your setup file. (Setting the prompt in noninteractive shells is pointless — and it can even cause problems (Section 4.5).)

—JP, TOR, and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As Section 4.1 explains, the simplest prompts — which I call static prompts — are prompts whose value are set once. The prompt doesn't change (until you reset the prompt variable, of course).

The default bash prompt is a good example of a static prompt. It's "bash$ " (with a space at the end, to make the command you type stand out from the rest of the prompt). You could set that prompt with the simple command:

PS1='bash$ '.

Notice the single quotes (Section 11.3) around the value; this is a good idea unless you want special characters in the prompt value to be interpreted before it's set. You can try it now: type that command on a command line, just as you would to set any other shell variable. Experiment a bit. The same prompt works on ksh and sh .

If you use csh or tcsh, try one of these, then experiment:

set prompt='csh% '

set prompt='tcsh> '

(zsh users: you can use any of the previous styles, but omit the set from the set prompt style.) Those prompts are fairly useless, right? If you log in to more than one machine, on more than one account, it's nice to have your hostname and username in the prompt. So try one of the following prompts. (From here on, I won't show a separate tcsh version with a > instead of a %. You can do that yourself, though, if you like.) If your system doesn't have uname, try hostname instead:

PS1="$USER@`uname -n`$ "

set prompt="$user@`uname -n`% "

Notice that I've used double quotes (Section 12.3) around the values, which lets the shell expand the values inside the prompt string before the prompt is stored. The shell interprets the variable $USER or $user — and it runs the command substitution (Section 28.14) that gives the hostname — once, before the prompt is set. Using double quotes is more efficient if your prompt won't change as you move around the system.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many shells can interpret the stored prompt string as each prompt is printed. As Section 4.1 explains, I call these dynamic prompts.

Special character sequences in the prompt let you include the current directory, date and time, username, hostname, and much more. Your shell's manual page should list these at the PS1 or prompt variable. (If you use the Korn shell or the original C shell, you don't have these special sequences. Section 4.4 has a technique that should work for you.)

It's simplest to put single quotes around the prompt string to prevent interpretation (Section 27.1) as the prompt is stored. For example, the following prompt shows the date and time, separated by spaces. It also has a special sequence at the end (\$ in bash, %# in tcsh and zsh) that's printed as a hash mark (#) if you're the superuser, or the usual prompt character for that shell otherwise. The first command in the following code listing works only in bash; the second only in tcsh:

PS1='\d \t \$ '                   ...bash

set prompt='%w %D %Y %P %# '      ...tcsh

PS1='%W %* %# '                   ...zsh

Having the history number in your prompt, as Section 4.14 shows, makes it easy to use history (Section 30.8) to repeat or modify a previous command. You can glance up your screen to the prompt where you ran the command, spot the history number (for example, 27), and type !27 to repeat it, !27:$ to grab the filename off the end of the line, and much more. In csh, tcsh, and bash prompts, use \! to get the history number. In zsh, use %! instead.

—JP, TOR, and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some shells don't have the special "dynamic" prompt-setting sequences shown in Section 4.3. If you still want a dynamic prompt, you probably can simulate one. Both ksh and bash will expand variables (like $PWD), do command substitution (to run a command like 'date'), and do arithmetic as they print the prompt. So, for example, you can put single quotes around the prompt string to prevent interpretation of these items as the prompt is stored. When the prompt string is interpreted, the current values will be put into each prompt. (zsh gives control over whether this happens as a prompt is printed. If you want it to happen, put the command setopt prompt_subst (Section 28.14) in your .zshrc file (Section 3.3).)

The following prompt stores the $PWD parameter to give the current directory, followed by a backquoted date command. The argument to date is a format string; because the format string is inside single quotes, I've used nested double quotes around it. Because it's in single quotes, it's stored verbatim — and the shell gets the latest values from date and $PWD each time a prompt is printed. Try this prompt, then cd around the filesystem a bit:

PS1='`date "+%D %T"` $PWD $ '

That prompt prints a lot of text. If you want all of it, think about a multiline prompt (Section 4.7). Or you could write a simple shell function (Section 29.11) named, say, do_prompt:

# for bash

function do_prompt {

   date=`date '+%D %T'`

   dir=`echo $PWD | sed "s@$HOME@~@"`

   echo "$date $dir"

   unset date dir

}



# for ksh

do_prompt( ) {

   date=`date '+%D %T'`

   dir=`echo $PWD | sed "s@$HOME@~@"`

   echo "$date $dir"

   unset date dir

}

and use its output in your prompt:

PS1='`do_prompt` $ '    ...for sh-type shells

The original C shell does almost no interpretation inside its prompt variable. You can work around this by writing a shell alias (Section 29.2) named something like setprompt ( Section 4.14) that resets the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Stray prompts can cause trouble for many commands that start a noninteractive shell. This problem may have (and probably has) been fixed in your C shell, but some of the following tricks will speed up your .cshrc, so keep reading.

If you set prompt in your .cshrc file without carefully checking first whether prompt was already set (Section 4.1), many older versions of the C shell will cheerfully print prompts into the pipe vi uses to expand glob characters, such as filename wildcards (*, ?, [ ]) (Section 1.13) and the tilde (~) (Section 31.11).

When you type :r abc*, vi opens a pipe to the C shell, writes the command

echo

abc*

down the pipe, then reads the response. If the response contains spaces or newlines, vi gets confused. If you set your prompt to ( n ) in your .cshrc [i.e., if you show the history number in parentheses as the prompt — TOR], vi tends to get:

(1) abc.file (2)

back from the C shell, instead of just abc.file.

The solution is to kludge your .cshrc like this:

if ($?prompt) then

    # things to do for an interactive shell, like:

    set prompt='(\!) '

endif

This works because a noninteractive shell has no initial prompt, while an interactive shell has it set to % .

If you have a large .cshrc, this can speed things up quite a bit when programs run other programs with

csh -c

'

command ', if you put all of it inside that test.

— CT

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

To set your prompt, you execute a command (on most shells, that command sets a shell variable). Before setting the prompt, you may run other commands to get information for it: the current directory name, for example. A shell can run two kinds of commands: built-in and external (Section 1.9). Built-in commands usually run faster than external commands. On a slow computer, the difference may be important — waiting a few seconds for your prompt to reset can get irritating (though the computer would have to be quite slow nowadays for it to matter that much). Creative use of your shell's built-in commands might pay off there, and they are still quite useful for those trying to squeeze the most performance out of their system. Let's look at some examples:

Section 4.3 has examples of some shells' special characters, such as %D to give the current date. Check your shell's manual page; if it has these features, using them won't slow you down the way an external command in backquotes (Section 28.14), like 'date', might.
If you're putting your current directory in your prompt, you may only want the tail of the pathname (the name past the last slash). How can you edit a pathname? You might think of using basename (Section 36.13) or sed (Section 34.1) with the current directory from $cwd — as in the first command in the following code listing, and probably in an alias like setprompt (Section 4.7) to make sure the prompt is updated whenever you change directories. The faster way is with the second command, using the C shell's built-in "tail" operator, :t:
```
set prompt="`basename $cwd`% "
```
{} Section 35.9
```
set prompt="${cwd:t}% "
```
If your current directory is /usr/users/hanna/projects, either of those prompts would look like "projects% " (with a space after the percent sign).
The C shell has several of these built-in string operators ( Section 28.5) like

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Lots of people like lots of information in their prompts: hostname, directory name, history number, and maybe username. Lots of people have spent lots of time trying to make their prompts short enough to fit across the screen and still leave room for typing a command longer than ls:

<elaineq@applefarm> [/usr/elaineq/projects/april/week4] 23 % ls

Even with fairly short prompts, if you look back at a screen after running a few commands, telling the data from the prompts can be a little tough (real terminals don't show user input in boldface, so I won't do it here either):

+<elaineq@applefarm> [~] 56% cd beta

<elaineq@applefarm> [~/beta] 57% which prog

/usr/tst/applefarm/bin/beta/prog

<elaineq@applefarm> [~/beta] 58% prog

61,102 units inventoried; 3142 to do

<elaineq@applefarm> [~/beta] 59%

Go to http://examples.oreilly.com/upt3 for more information on: mlprompt.cshmlprompt.sh

One nice answer is to make a prompt that has more than one line. Here's part of a .cshrc file that sets a three-line prompt: one blank line, one line with the hostname and current directory, and a third line with the history number and a percent sign. (If this were a tcsh, I could have gotten the hostname with %m.) The C shell quoting (Section 27.13) is ugly — doubly ugly because the prompt is set inside an alias — but otherwise it's straightforward:

uname -n Section 2.5, {..} Section 35.9

set hostname=`uname -n`

alias setprompt 'set prompt="\\

${hostname}:${cwd}\\

\! % "'

alias cd 'chdir \!* && setprompt'

alias pushd 'pushd \!* && setprompt'

alias popd 'popd \!* && setprompt'

setprompt           # to set the initial prompt

(There's a version on the Web for Bourne-type shells.) The prompts look like this:

applefarm:/usr/elaineq/projects/april/week4

23 % prog | tee /dev/tty | mail -s "prog results" bigboss@corpoffice

61,102 units inventoried; 3142 to do



applefarm:/usr/elaineq/projects/april/week4

24 % cd ~/beta



applefarm:/usr/elaineq/beta

25 % prog | mail joanne

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some people don't like to put the current directory, hostname, etc. into their prompts because it makes the screen look cluttered to them. Here's another idea. If your terminal or window system has a status line or titlebar, you might be able to put the information there. That's nice because you can see the information while you run programs. The down side is that the information can get out of date if you use a command that takes you to another host or directory without updating the status line. The latest bash and zsh shells do this by default when you're using an xterm window. For the rest of you, here's how to do it yourself. Because neither csh or tcsh do this by default, I'll show C-shell-type syntax. But you can do the same thing in Bourne-type shells with a shell function and case (Section 35.10) statement; there's a ready-to-use version on the web site.

When you use cd, pushd, or popd, an alias uses the echo command to write special escape sequences to the terminal or window.

Here are cd aliases and other commands for your .cshrc or .tcshrc file. If I were logged in to www.jpeek.com in the directory /home/jpeek, this alias would put:

www:/home/jpeek

in the status area or window title, depending on which terminal type I'm using. Of course, you can change the format of the status line. Change the following command string, ${host:h}:${cwd}, to do what you need; see your shell's manual page for a list of variables, or create your own custom information.

:h Section 28.5, && Section 35.14

Go to http://examples.oreilly.com/upt3 for more information on: stattitle.cshstattitle.sh

set e=`echo x | tr x '\033'`   # Make an ESCape character



set g=`echo x | tr x '\07'`    # And a Ctrl-g

set host=`uname -n`

# Puts $host and $cwd in VT102 status line. Escape sequences are:

# ${e}7 = save cursor position, ${e}[25;1f = go to start of status

# line (line 25), ${e}[0K = erase line, ${e}8 = restore cursor

alias setstatline 'echo -n "${e}7${e}[25;1f${e}[0K    ${host:h}:${cwd}${e}8"'

alias settitle 'echo -n "${e}]2;${host:h}:${cwd}${g}"'

switch ($TERM)

case vt10?:

  alias cd 'cd \!* && setstatline'

  alias pushd 'pushd \!* && setstatline'

  alias popd 'popd \!* && setstatline'

  breaksw

case xterm*:

  alias cd 'cd \!* && settitle'

  alias pushd 'pushd \!* && settitle'

  alias popd 'popd \!* && settitle'

  breaksw

endsw

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some people don't want to be faced with a Unix % or $ shell prompt. If you (or, if you're a sys admin on a multiuser system, your users) usually run only a few particular Unix commands, you can put those command names in the shell prompt. Here's a simple one-line Bourne-shell prompt for a .profile:

PS1='Type "rn", "mailx", "wp", or "logout": '

Next, a multiline prompt (Section 4.7) for the C shell .cshrc or .tcshrc file:

if ($?prompt) then

set prompt='\\

Type "pine" to read the news,\\

type "mutt" to read and send mail,\\

type "wp" for word processing, or\\

type "logout" to log out.\\

YES, MASTER? '

endif

You get the idea.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If your prompt has some information that you want to stand out — or if you want your whole prompt to stand out from the rest of the text on the screen — you might be able to make it in enhanced characters or colors. If your terminal has special escape sequences for enhancing the characters (and most do), you can use them to make part or all of your prompt stand out. Newer versions of xterm also have color capability, as does the Mac OS X Terminal program, though Terminal may not properly support the escape sequences we discuss later. (The GNU dircolors (Section 8.6) command sets up a color-capable terminal.)

Go to http://examples.oreilly.com/upt3 for more information on: blinkprompt.cshblinkprompt.sh

Let's say that you want to make sure people notice that they're logged in as root (the superuser) by making part of the root prompt flash. Here are lines for the root .cshrc:

# Put ESCape character in $e.  Use to start blinking mode (${e}[5m)

# and go back to normal mode (${e}[0m) on VT100-series terminals:

set e="`echo x | tr x '\033'`"

uname -n Section 2.5

set prompt="${e}[5mroot${e}[0m@`uname -n`# "

That prompt might look like this, with the word root flashing:

root@www.jpeek.com#

Shells with command-line editing need to calculate the width of your prompt string. When you put nonprinting escape sequences in a prompt (as we're doing here), in zsh and tcsh you have to delimit them with %{ and %}. In bash , bracket nonprinting characters with \[ and \]. In the Korn shell, prefix your prompt with a nonprinting character (such as CTRL-a) followed by a RETURN, and delimit the escape codes with this same nonprinting character. As the pdksh manual page says, "Don't blame me for this hack; it's in the original ksh."

The prompt is set inside double quotes (

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Both zsh and tcsh have an optional prompt at the right side of the screen. Unlike the normal left-side prompt, the cursor doesn't sit next to the right-side prompt (though the right prompt disappears if you type a long command line and the cursor passes over it). It's stored in the zsh RPROMPT variable and in tcsh rprompt.

What can you do with a right-hand prompt? Anything you want to! (You'll probably want to keep it fairly short, though.) Put the time of day on the right-hand side, for instance; on tcsh, it's this easy:

[jpeek@ruby ~]% set rprompt='%t'

[jpeek@ruby ~]% users                                         3:44pm

jpeek ollie

[jpeek@ruby ~]%                                               3:45pm

As another idea, you could use sched to remind you of an important meeting by setting the right-hand prompt. Here's a shell function for zsh that sets the right prompt to "LEAVE NOW" at a particular time. You can give it one argument to set the time to remind you. Or, with no argument, it removes the right-hand prompt:

leave( ) {

    case "$#" in

    0) unset RPROMPT ;;

    1) sched "$1" "RPROMPT='LEAVE NOW'" ;;

    *) echo "Usage: leave [time]" 1>&2 ;;

    esac

}

Here's an example:

jpeek$ date

Fri May 12 15:48:49 MST 2000

jpeek$ leave 15:55

               ...do some work...

jpeek$ pwd

/u/jpeek/pt

jpeek$ date                                        LEAVE NOW

Fri May 12 15:55:22 MST 2000

jpeek$ lpr report                                  LEAVE NOW

jpeek$ leave                                       LEAVE NOW

jpeek$

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're like me, when you start a shell escape (Section 17.21) or any subshell (Section 24.4), you can forget that you aren't in your login shell. Your shell history (Section 30.1) might get confused, shell variables (Section 35.9) may not be set, and other problems may come up. zsh and bash have a built-in SHLVL environment variable (Section 35.3) that lets you track how many subshells deep your current shell is. tcsh has a shlvl shell variable that's automatically set from (and sets) SHLVL. So, all three shells cooperate with each other to set the right value, even if you start one shell from another. (For other shells that don't have SHLVL — ksh and csh — you can set up something similar with a bit of arithmetic in the ENV (Section 35.5) file or the .cshrc file, respectively.)

In your top-level shell, the value of $shlvl is 1 (one). In the first subshell, it's 2; in a sub-subshell, it's 3; and so on. You can use this to control your shell startup files — for example, have some commands in your .cshrc that run when you first log in (and $shlvl is 1), but don't run in subshells. You can also put $shlvl in your prompt (but only during subshells, if you'd like — as a reminder that you aren't in your top-level shell). You can set your prompt to mike% in top-level shells, (1) mike% in a first-level subshell, (2) mike% in a second-level subshell, and so on. Here's some sample prompt-setting code for your .tcshrc:

# If this is a subshell, put shell level in prompt:

if ($shlvl == 1) then

    set prompt="${USER}% "

else

    set prompt="($SHLVL) ${USER}% "

endif

bash doesn't need an if because login shells read your .bash_profile (or .profile) and subshells read your .bashrc. Here are commands to set the prompts I mentioned earlier:

PS1='\u\$ '             ...for the .bash_profile

PS1='($SHLVL) \u\$ '    ...for the .bashrc

Does your account run a windowing system that's started from your top-level shell startup file (like

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This tip is also great if you use a mouse to copy and paste command lines in your window.

Some terminals I've used (like old Hewlett-Packard and Tektronix terminals) had local editing. You could move your cursor up the screen to a previous command line, maybe make some edits to it, then press a SEND LINE key to resend that line to the host. This didn't have anything to do with sophisticated command-line editing (Section 30.14) that modern Unix shells have, though. Maybe your terminal can do that, too. Depending on how your emacs editor is configured, shell-mode may work that way, as well.

The problem was that unless I erased the shell prompt (%) on my screen, it would be sent back to the shell and give the error "%: Command not found." So I set my shell prompt to this:

set prompt='     '

That's right: four spaces. Most Unix commands start their output at column 1, so my command lines were easy to find because they were indented. The shell didn't care if I sent four spaces before the command line. So everything was fine until I got my new terminal without a SEND LINE key . . .

If you want some information in your prompt, too, make a multiline prompt (Section 4.7) with four spaces in the last line.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many people use the current directory in their prompts. If you use the pushd and popd (Section 30.7) commands, you may not always remember exactly what's in your directory stack (I don't, at least). Here's how: run the dirs command, and use its output in your prompt. A simple csh and tcsh alias looks like this:

alias cd 'chdir \!* && set prompt="`dirs`% "'

and the prompts look like:

/work/project % cd

~ % cd bin

~/bin %

Here's what to put in .cshrc or .tcshrc to make a multiline prompt (Section 4.7) that shows the directory stack:

uname -n Section 2.5, expr Section 36.21

# PUT hostname.domain.name IN $hostname AND hostname IN $HOST:

set hostname=`uname -n`

setenv HOST `expr $hostname : '\([^.]*\).*'`

alias setprompt 'set prompt="\\

Go to http://examples.oreilly.com/upt3 for more information on: dirs-prompt.cshdirs-prompt.sh

${USER}@${HOST} `dirs`\\

\! % "'

alias cd  'chdir \!* && setprompt'

alias pushd  'pushd \!* && setprompt'

alias popd  'popd  \!* && setprompt'

setprompt   # SET THE INITIAL PROMPT

Because bash can run a command each time it sets its prompt, and because it has built-in prompt operators (Section 4.3) like \u, the bash version of all the previous stuff fits on one line:

$(...) Section 28.14

PS1='\n\u@\h $(dirs)\n\! \$ '

That makes a blank line before each prompt; if you don't want that, join the first and second lines of the setprompt alias or remove the first \n. Let's push a couple of directories and watch the prompt:

jerry@ora ~

1 % pushd /work/src/perl

/work/src/perl ~



jerry@ora /work/src/perl ~

2 % cd ../cnews



jerry@ora /work/src/cnews ~

3 % pushd ~/bin

~/bin /work/src/cnews ~



jerry@ora ~/bin /work/src/cnews ~

4 %

Of course, the prompt looks a little redundant here because each pushd command also shows the dirs

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Bourne shell's trap (Section 35.17) will run one or more commands when the shell gets a signal (Section 24.10) (usually, from the kill command). The shell will run any command, including commands that set shell variables. For instance, the shell could reread a configuration file; Section 24.13 shows that. Or it could set a new PS1 prompt variable that's updated any time an external command (like another shell script or a cron job (Section 25.2)) sends the shell a signal. There are lots of possibilities.

This trick takes over signal 5 (SIGTRAP), which usually isn't used. When the shell gets signal 5, a trap runs a command to get the date and time, then resets the prompt. A background (Section 23.2) job springs this trap once a minute. So, every minute, after you type any command, your prompt will change.

You can use any command's output in your prompt (possibly with some editing, probably with sed (Section 34.1) or expr (Section 36.21)): count the number of users, show the load average (Section 26.4), whatever. Newer shells, like bash, can run a command in backquotes (Section 28.14) each time the prompt is displayed — Section 4.10 has an example. But, to have an external command update a shell variable at any random time, this trap trick is still the best.

Go to http://examples.oreilly.com/upt3 for more information on: date- prompt.sh

Now on to the specific example of putting date and time in the old Bourne shell's prompt. If your system's date command doesn't understand date formats (like +%a), get one that does. Put these lines in your .profile file (or just type them in at a Bourne shell prompt):

# Put date and time in prompt; update every 60 seconds:

trap 'PS1=`date "+%a %D %H:%M%n"`\

$\ ' 5

while :

: Section 36.6

do

    sleep 60

    kill -5 $$

done &

promptpid=$!

Now, every minute after you type a command, your prompt will change:

Thu 06/20/02 02:33

$ cc bigprog.c

undefined symbol                first referenced in file

xputc                               bigprog.o

ld fatal: Symbol referencing errors.

Thu 06/20/02 02:34

$

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

bash, tcsh, and zsh can run a Unix command, or multiple commands, before printing each prompt. tcsh and zsh also can do something you specify before executing the command you've typed at a prompt. Finally, tcsh and zsh can do something periodically (every n seconds) before whatever prompt comes next. (Section 4.15 shows how to execute commands periodically in the original Bourne shell.) These commands don't have anything to do with setting the prompt itself, though they can. The command could do some system checking, reset shell variables, or almost anything that you could type at a shell prompt. If the commands run slowly, they'll delay whatever else you're doing, so keep that in mind.

Let's start with precmd , the tcsh alias that's run after your command line is read and before the command is executed. In zsh, the same thing is done by the shell function named preexec. Shell history is available, so you can use history substitution (Section 30.8) inside the alias or function. Here's a nice example adapted from the tcsh manual page: showing the command line you're running in your xterm window titlebar. It's ugly because it has ESC and CTRL-g characters embedded directly in the alias; I'd rather store the escape sequences in shell variables, as shown in the xterm titlebar article (Section 4.8). The if sets the alias only if you're using an xterm terminal:

# Show each command line in xterm title bar:

if ($TERM == xterm) alias postcmd 'echo -n "^[ ]2;\!#^G"'

Next, let's look at running a command periodically. You'd like to watch the load average by running uptime (Section 26.4) every minute, before a prompt. Here's how to do it in zsh: put code like this in your .zshrc file (Section 3.3) (or just type it at a prompt to try it). The PERIOD shell variable is the interval, in seconds, between runs of the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Is there something you want to do every time you log out: run a program that deletes temporary files, asks you a question, or prints a fortune to your screen? If you use the C shell, make a file named .logout (Section 3.3) in your home directory and put the commands there. Before a login C shell exits, it will read that file. A login bash reads .bash_logout, and zsh reads .zlogout. But not all shells are login shells; you might want these shells to read your logout-type file, too. Section 3.18 shows a fix for the Bourne and Korn shells; Section 3.8 and Section 3.4 have background information.

Some ideas for your logout file are:

A command like fortune to give you something fun to think about when you log out.
A command to list a "reminder" file — for example, work to take home.
A script that prompts you for the hours you've worked on projects so you can make a timesheet later.
The command clear to erase your screen. This keeps the next user from reading what you did. In the Mac OS X Terminal application, command-k will delete the scrollback buffer. It also helps to stop "burn-in" damage to old, monochrome monitors caused by characters left over from your login session (though this is hardly a concern nowadays; most of us have moved on to color screens that are not subject to burn-in). (Some Unixes clear the screen before printing the login: prompt. Of course, this won't help users who connect with a data switch or port manager because the connection will be broken before the next login prompt.)

If you connect to this host over a network, with a slow modem or on a data switch — and you don't see all the logout commands run before your connection closes — try putting the command sleep 2 (Section 25.9) at the end of the file. That makes the shell wait two seconds before it exits, which gives output more time to get to your screen.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 4.17 describes logout files. Commands in those files are run when you log out. The Bourne and Korn shells don't have a logout file, though. Here's how to make one:

In your .profile file, add the line:
trap Section 35.17, . Section 35.29
```
trap '. $HOME/.sh_logout; exit' 0
```
(Some systems may need $LOGDIR instead of $HOME.)
Make a file in your home directory named .sh_logout. Put in the commands you want to be run when you log out. For example:
if Section 35.13, [ -f Section 35.26
```
clear

if [ -f $HOME/todo.tomorrow ]

then

    echo "=========== STUFF TO DO TOMORROW: ============"

    cat $HOME/todo.tomorrow

fi
```

The trap will read the .sh_logout file when the shell exits.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It's pretty easy to type one too many CTRL-d characters and log out of a Bourne shell without meaning to. The C shell has an ignoreeof shell variable that won't let you log out with CTRL-d. So do the Korn shell and bash; use set -o ignoreeof.

Here's a different sort of solution for the Bourne shell. When you end the shell, it asks if you're sure. If you don't answer yes, a new shell is started to replace your old one.

First, make a file like the C shell's .logout that will be read when your Bourne shell exits (Section 4.18). Save your tty (Section 2.7) name in an environment variable (Section 35.3), too — you'll need it later:

trap Section 35.17

TTY=`tty`; export TTY

trap '. $HOME/.sh_logout; exit' 0

(Your system may need $LOGDIR instead of $HOME.) Put the following lines in your new .sh_logout file:

exec < Section 36.15, case Section 35.11, exec Section 24.2, -sh Section 3.19

exec < $TTY

echo "Do you really want to log out? \c"

read ans

case "$ans" in

[Yy]*) ;;

*)  exec $HOME/bin/-sh ;;

esac

The last line uses some trickery to start a new login shell (Section 3.19). The shell closes your tty (Section 36.15) before reading your .sh_logout file; the

exec <

$TTY

reconnects the shell's standard input to your terminal.

Note that if your system is very slow, you may not get the reminder message for a couple of seconds — consequently, you might forget that it's coming and walk away. That hasn't been a problem where I've tested this. If it is for you, though, replace the read ans with a program like grabchars that times out and gives a default answer after a while. There may be some Bourne shells that need other tricks — and others that don't need these tricks — but this should give you an idea of what to do.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This chapter covers most of what you need to know to set up your terminal or terminal emulator from your shell setup files (Section 3.3).

In the latter half of the chapter, we cover the ins and outs of working with some of the most popular terminal-emulator software for the X Window System, including xterm, rxvt, and others, where applicable. The list of terminals and emulators you might come into contact with is long and getting longer, though, so the advice we give in the first section of the chapter regarding how to configure your terminal will be helpful. As you find yourself suddenly confronted with the prospect of configuring the terminal emulator on your cell phone or tablet computer, remember: you can usually make it work, with enough time and effort.

It is important to remember, however, that the tricks and tips we discuss in this chapter, if implemented incorrectly, may cause your terminal to hang. One way around a hung terminal is always to keep at least one other terminal emulator window, with sane settings, open all the time you're modifying the setup of the other. That way, if you hang up the terminal you're actively modifying, you can always go back to the other and save yourself. On systems that support virtual consoles, such as Linux, you can also use command keys (e.g., ALT and the first five function keys) to switch between various virtual consoles, just as you might with a terminal emulator. Don't just reach for the power switch!

—TOR and SJC

In the past few years, terminals have been standardized to a few types. In fact, most terminals nowadays are terminal emulators (like xterm) that simulate a terminal on a graphical display. Years ago, though, terminals differed widely. Rather than simply being implemented in software, they were hardware — keyboards and monitors or even teletypes, with which the user interacted to communicate with an often faraway mainframe or other big iron. All were specialized, and differences between them often came down to how much you paid and to what manufacturer. This lets you take advantage of other features of the manufacturer's primary hardware — the big computers they considered their main product. Manufacturers produced a variety of terminals, each one including a particular set of features for a certain price. There were smart terminals and dumb ones, terminals with big screens and terminals with small screens, printing terminals and video displays, and terminals with all sorts of special features.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This chapter covers most of what you need to know to set up your terminal or terminal emulator from your shell setup files (Section 3.3).

In the latter half of the chapter, we cover the ins and outs of working with some of the most popular terminal-emulator software for the X Window System, including xterm, rxvt, and others, where applicable. The list of terminals and emulators you might come into contact with is long and getting longer, though, so the advice we give in the first section of the chapter regarding how to configure your terminal will be helpful. As you find yourself suddenly confronted with the prospect of configuring the terminal emulator on your cell phone or tablet computer, remember: you can usually make it work, with enough time and effort.

It is important to remember, however, that the tricks and tips we discuss in this chapter, if implemented incorrectly, may cause your terminal to hang. One way around a hung terminal is always to keep at least one other terminal emulator window, with sane settings, open all the time you're modifying the setup of the other. That way, if you hang up the terminal you're actively modifying, you can always go back to the other and save yourself. On systems that support virtual consoles, such as Linux, you can also use command keys (e.g., ALT and the first five function keys) to switch between various virtual consoles, just as you might with a terminal emulator. Don't just reach for the power switch!

—TOR and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In the past few years, terminals have been standardized to a few types. In fact, most terminals nowadays are terminal emulators (like xterm) that simulate a terminal on a graphical display. Years ago, though, terminals differed widely. Rather than simply being implemented in software, they were hardware — keyboards and monitors or even teletypes, with which the user interacted to communicate with an often faraway mainframe or other big iron. All were specialized, and differences between them often came down to how much you paid and to what manufacturer. This lets you take advantage of other features of the manufacturer's primary hardware — the big computers they considered their main product. Manufacturers produced a variety of terminals, each one including a particular set of features for a certain price. There were smart terminals and dumb ones, terminals with big screens and terminals with small screens, printing terminals and video displays, and terminals with all sorts of special features.

Differences between terminals do not matter much to programs like cat (Section 12.2) or who (Section 2.8) that use the terminal screen as a sort of typewriter with an endless scroll of paper. These programs produce sequential output and do not make use of the terminal's special features; they do not need to know much to do their job. Only programs such as screen editors, which make use of screen-handling features, need to know a lot about differences between terminals.

However, even today, we find a wide variety of terminal emulators across a multitude of platforms. My new Kyocera Smartphone, for example, is a Palm device integrated with a PCS telephone; one of the main reasons I bought it was for remote, emergency ssh access to my servers, using a tiny terminal emulator that runs on the PalmOS. Many Unix programs assume a basic environment that this terminal emulator does not provide — an 80-column screen — so even simple commands such as w, which prints a list of who is logged in, where they logged in from, and what they're currently running, become impossible to run. But let's go back to the early days and revisit some of the old problems that plagued early Unix developers, so that we might better understand how to deal with today's problems.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you always work at the same terminal or use the same terminal emulator, there's no problem with setting the terminal type explicitly in your shell setup file (Section 3.3) — like .login or .profile. Just set the TERM environment variable (Section 35.3):

setenv TERM vt100           ...csh, tcsh

TERM=vt100; export TERM     ...sh, ksh, zsh

export TERM=vt100           ...pdksh, bash, zsh

In fact, on a hardwired terminal, your terminal type may already have been set in a system file like /etc/ttys or /etc/ttytype (Section 3.15). But if, like many Unix users, you might log in from time to time at different terminals, from home, or on different systems over a network, you may need some more intelligent method for setting the terminal type. To find out, try logging in at each place and starting a screen-oriented program like vi. Do various operations: scrolling up, inserting text that wraps onto another line, deleting lines. If the screen scrambles or the cursor gets "out of sync," your terminal type may need to be set.

It's possible to set up various tests (Section 3.10) in your shell setup files to do this. But you can also do a surprising amount of terminal type testing with tset, even though it was nominally designed for initializing the terminal:

If no arguments (Section 1.4) are specified and TERM is already set, tset uses the value of TERM to determine the terminal type.
If no arguments are specified and TERM is not set, then tset uses the value specified in the system file /etc/ttytype or /etc/ttys (BSD 4.3 and later and its derivatives only). On Linux systems, the terminal type is determined by getty, based on a similar process but using the /etc/inittab file or other configuration files used by getty during initialization. On SVR4 systems, a similar process is managed by ttymon and listen.
If a terminal type is specified as an argument, that argument is used as the terminal type, regardless of the value of TERM.
The -m (map) option allows a fine degree of control in cases where the terminal type may be ambiguous. For example, if you sometimes log in on a dialup line, sometimes over a local area network, and sometimes on a hardwired line, the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

tset (Section 5.3) is a powerful tool to use if you often log in at different terminals. You can use tset to prompt you with a default terminal type, giving you the opportunity to specify a new terminal type when you log in:

TERM = (vt100)

However, tset requires you to know your terminal type. You might log in at a new terminal and have no idea what to set the terminal type to. Or your terminal might be configured to emulate another terminal type without your knowledge. New users in particular are confused by the tset prompt. In some respects, this is not a surprise, as the prompt itself can be confusing without a bit of context.

Go to http://examples.oreilly.com/upt3 for more information on: qterm

As an alternative, try Michael Cooper's qterm program on our CD-ROM [see http://examples.oreilly.com/upt3]. qterm sends the terminal a test string and determines what sort of terminal you're using based on how the terminal responds. Using qterm, you can make sure you always use the correct terminal type by placing the following line in your .login :

'...' Section 28.14

setenv TERM `qterm`

or in .profile :

TERM=`qterm`;export TERM

The advantage of qterm is that it sets the terminal type without your intervention. You don't need to know your terminal type; it gets set automatically.

qterm works by sending the terminal a query string and returning the terminal type depending on the terminal's response. qterm is configured using a listing of responses and the terminals to which they correspond. By default, qterm looks for the listings in a system-wide location such as /usr/local/lib/qtermtab. In addition, you can call qterm with the + usrtab option, so that it will look for a file called .qtermtab in your home directory.

The string used to query the terminal is usually ESC Z. The sample qtermtab file distributed with qterm defines the responses several different terminals give for that string:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When the xterm client is called, it not only sets the TERM environment variable, but it also adjusts the terminal definition for the size of the window being created. The size of xterm windows, however, can be changed later on by using the window manager. If the window is resized, then the user's shell may need to be passed the new size information as well, or programs that use termcap and terminfo won't work correctly. The resize client is provided for redefining the number of lines and columns for the terminal database used in an xterm window. Note that resize cannot be used for terminal emulators other than xterm (except for those, like rxvt, that emulate xterm) because it depends on xterm's escape sequences.

Some systems can send a "window size changed" signal (SIGWINCH) to programs and do not require resize to be run for a resized xterm window. We recommend using resize only if terminal-based programs start to have problems with your window size. A typical terminal-based program that is having problems with the window size will fill only some of the lines in the window — or may scroll lines down the window when it shouldn't.

The resize client is typically used immediately after the dimensions of an xterm window are changed. A peculiarity of the resize client is that it does not access the shell itself, but simply returns the shell commands that would be needed; to have those commands read by the shell, you either save its output in a file and read the file with the shell commands source or . (Section 35.29), or evaluate resize output using the shell command eval (Section 27.8). For example, after resizing a window, you would type in that shell:

`...` Section 28.14

% eval `resize`

When you call the resize command under a termcap system, it produces the commands for resetting the TERMCAP environment variable with the li# and co# capabilities reflecting the current dimensions. When you call the resize command under a terminfo system, it produces the commands for resetting the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If your terminal seems to "hang" (freeze, lock up) when you log in, here are some things to try:

Have another user look at your shell's setup files (Section 3.3). There could be some obvious mistakes that you didn't catch.
Log in to another account and use the su stucklogin command (if the stuck account uses Bourne-type shells) or the su -f stucklogin command (if the stuck account uses csh or tcsh). Change (cd ) to the home directory. Rename the account's setup files so the shell won't see them as you log in. (If you have superuser access (Section 1.18), you also can use it to rename the file.)
If you can log in after that, you know that the problem is with the account's setup files.
Set shell debugging ( Section 27.15) on the stuck account's setup files. From another account or as the superuser, start an editor and put the following line at the top of an sh-like setup file (such as .profile). It'll tell you whether .profile is being read at all and where it hangs:
```
set -xv
```
You'll see each line read from the .profile and the commands executed on the screen. If you don't see anything, then the shell probably didn't read .profile. Bash users would want to check .bashrc or .bash_profile.
C-shell users should put this command at the top of .cshrc (or .tcshrc, for tcsh) instead:
```
set echo verbose
```
Note that on many Unix systems, the shell won't read its startup files if the files aren't owned by you. You might use ls -l (Section 50.2) to check.
Look at the entry in the /etc/passwd file (Section 22.3) for this user. Be sure it has the correct number of fields (separated by :). Also, see if there's another user with the same login name. (If your system has commands like

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: stty

It may hardly seem appropriate to follow Chris Torek's learned article about how stty works with some basics, but this book is designed for beginners as well as those who already know everything. :-) [Good idea, Tim. This is also a handy place to put the globe icon for the GNU version. ;^) — JP]

So, to find out what settings your terminal line currently has, type:

% stty

For a more complete listing, type:

% stty -a

On older BSD-style systems, use stty everything instead. On most newer BSD-derived systems,

stty

everything

and stty -a are both supported, but with slightly different output formats. The former prints a tabular layout, while the latter prints each control character setting in a name = value format.

As Jerry Peek said in an editorial aside to Chris's article, be sure to have your stty manual page handy!

—TOR and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Have you ever sat down at a terminal where the "erase" key (the character that deletes the last thing you typed) wasn't where you thought it would be? If you have, you know how disorienting this can be! On Linux, there's loadkeys. If you're using the X Window System, check into the xmodmap (Section 5.1) command. Newer shells, like bash and zsh, tend to do their own handling of these special characters — especially during their built-in command-line editing (Section 30.14). Check your shell's manual page about readline . The most portable method is with the stty (Section 5.7) command. All of these give you a way of changing the erase character (along with several others) so you can restore some order to your world.

stty takes two kinds of input. If you want to give the command interactively, type stty erase char, where char is the key you normally use for erase — BACKSPACE, DELETE, whatever — followed by RETURN. This will do the trick, provided that the character you type isn't already used for something. If the character is in use or if you're putting stty commands into your .login, .profile, or .bash_profile file, it's better to "spell these characters out." "Control" characters in .login are allowed, but they aren't a great idea. If you like to use the BACKSPACE key as the erase key, add the following line:

stty erase ^h

If you want to use the DELETE key, quote the ? character so the shell won't treat it as a wildcard (Section 1.13):

stty erase ^\?

That is, stty lets you represent a control key with the two-character combination ^ x, where ^ is the literal key ^ (caret) and x is any single character. You may need to put a \ before the x to prevent the shell from interpreting it as a wildcard [and a \ before the ^ to prevent old Bourne shells from interpreting it as a pipe! — JP].

Of course, you're not limited to the BACKSPACE or DELETE keys; you can choose any other key you want. If you want to use "Z" as your DELETE key, type

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

xterm is by far the most commonly used X client, although more and more people are switching from xterm to similar or related programs, such as rxvt — which is a lightweight xterm derivative without the Tektronix terminal emulation support. Regardless, the most commonly used clients are largely derivatives of xterm, so we're devoting the rest of this section to this single client and its family.

xterm gives you a window containing your standard shell prompt (as specified in your /etc/passwd entry). You can use this window to run any command-line-oriented Unix program or to start additional X applications.

The uncustomized xterm window should be sufficient for many users' needs. Certainly you can do anything in a vanilla xterm window that you can from a character-based terminal. But xterm also has special features you can use, and since you spend so much time in xterm, you might as well use them.

The rest of this chapter gives you a set of tricks and tips about using xterm, including the following:

Specifying and using a scrollbar (Section 5.11).
Copying and pasting text selections (Section 5.13).
Modifying text-selection behavior (Section 5.14).
Printing the current directory in the xterm titlebar (Section 5.15).
Dynamically changing fonts and other features (Section 5.17, Section 5.18).

The articles in this chapter use terms that you may want defined:

A pointer , or pointing device, is a piece of hardware designed for navigating a screen. Most people use a mouse as their pointer, but there are also trackballs, touchpads, and others.
The best pointer to use with X has three buttons. When we refer to the first button or button 1, we mean the button you click with your index finger. For right-handed people, this is usually the left button on a mouse. But the X client xmodmap (Section 6.1) lets left-handed users swap mouse buttons to make the rightmost button the "first."
Even though the actual image on the screen is called a cursor, throughout this chapter we refer to "moving the pointer" to avoid confusion with the standard text cursor that can appear in an

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you want your xterm or rxvt to run a login shell (Section 3.4), give it the -ls flag, or put a line like one of the following in your X resource file (Section 6.5):

xterm*loginShell: true  ...for xterm

XTerm*loginShell: true  ...for xterm or rxvt

Rxvt*loginShell:  true  ...for rxvt

Once you've defined the appropriate resource, you can get a nonlogin shell (which is otherwise the default) with xterm +ls.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The scrollbar is a favorite xterm feature, particularly among those whose terminals lacked the ability to scroll backwards. Using the scrollbar, you can re-examine the output or error from a command, select previous text to supply in another command line or to paste into a file, or to hide your current screen from a nosy coworker.

There are many ways to start up the scrollbar. You can specify the -sb option on the command line:

% xterm -sb &

% rxvt -sb &

or you can set the scrollBar resource (Section 6.5) to true:

XTerm*scrollBar:    true  ...for xterm or rxvt

Rxvt*scrollBar:            true  ...for rxvt

or for an xterm window that's already running, you can call up the VT Options menu (Section 5.17) by holding down the CTRL key and the center mouse button or by selecting Enable Scrollbar. These menus are not supported by rxvt.

A scrollbar appears on the left side of the xterm window, as shown in Figure 5-1.

Figure 5-1: xterm window with scrollbar

—LM and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you use the scrollbar in xterm (Section 5.11), you'll find that by default the scrollbar retains only 64 previous lines of text. You can change this by using the -sl command-line option:

% xterm -sb -sl 200 &

% rxvt -sb -sl 200 &

or by setting the saveLines resource:

XTerm*saveLines: 200

You don't want to go crazy with the number of saved lines, though. Too many lines saved may crunch on virtual memory and also make it hard to scroll.

—LM and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can use the pointer to select text to copy and paste within the same xterm window or between xterm windows. You don't need to be in a text editor to copy and paste. You can also copy or paste text to and from the command line, between the command line and a file, etc.

There are several ways to select (copy) text; all require you to use the pointer. You can select a passage of text, or you can select text by individual words or lines.

When you select text, it is highlighted and copied into global memory from which you can paste it into any xterm window. Regardless of the number of xterm windows you're running, you can store only one selection in memory at a time. However, you can paste that selection as many times as you like. When you make another selection, the new text replaces the previous selection in memory.

Table 5-2 summarizes all of the text-selection methods.

Table 5-2: Button combinations to select text for copying
To select	Do this
Passage	Click the first button at the start of the selection and the third button at the end of the selection. Or at the beginning of the selection, hold down the first button; drag the pointer to the end of the desired text; release the button.
Word	Double-click the first button anywhere on the word.
Line	Triple-click the first button anywhere on the line.

To clear the highlighting, move the pointer off the selection, and click the first button anywhere else in the window. Note, however, that the text still remains in memory until you make another selection.

Of the two methods for selecting a passage, the first is generally easier. Hypothetically, you can select a passage of any length; in practice, we've found there to be limitations. The size of the window limits the amount of text you can highlight in one action. You can extend a selection beyond the parameters of a window. Copying an extremely long selection, however, doesn't seem to work reliably. Also, when pasting a long selection, the text can become garbled.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You probably already know how to select text (Section 5.13) in an xterm, and you've probably discovered that double-clicking Section 5.13 will select the entire word around the pointer. What you may not know is that it is possible to change what defines a "word."

xterm maintains a table of all the ASCII characters and their character classes. Any sequence of adjacent characters of the same class is treated as a word. Numbers, letters, and the underscore are in class 48 (which is the ASCII code for the character 0) and SPACE and TAB are in class 32 (the ASCII code for SPACE). By default, all the other characters are in classes by themselves.

For Unix users, this isn't the most useful default; it would be better if you could select filenames, email addresses, URLs, resource specifications, etc. as single words even though they often contain punctuation characters.

You can modify the character class table with xterm's charClass resource variable (Section 6.3). The value this resource accepts is a comma-separated list; each item on the list is an ASCII character code or range of characters, followed by a colon, followed by the character class to which the character should be added. I set the charClass resource as follows:

xterm*charClass: 33:48, 37:48, 42:48, 45-47:48, 63-64:48, 126:48

This tells xterm to treat !, %, *, -, ., /, ?, @, and ~ as characters of the same class as numbers and letters. You may also want to treat : as a member of this class, for URLs; in that case, use the following charClass string:

xterm*charClass: 33:48, 37:48, 42:48, 45-47:48, 58:48, 63-64:48, 126:48

—DJF and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Under most modern window managers, most windows (including xterm) are displayed with a titlebar. You can change the text in the titlebar using the following xterm escape sequence:

^[ ]2;string^G

Note that this sequence has a close bracket (]) following the ESC (Escape, ^[) — not an open bracket. It ends with a CTRL-g character — not a caret followed by a "g".

I use this sequence to display my current working directory and directory stack in the titlebar, where they are visible but unobtrusive. I do this by adding a few lines to my shell setup file (Section 3.3). Section 4.8 explains.

If you change the number "2" in the escape sequence to "1," it will set the text that appears in the xterm's icon instead of its titlebar. If you change it to "0," it will set the text for both the icon and the titlebar. If you use and iconify a number of xterms, you may find these sequences useful.

You may also wish simply to specify an icon name and/or title text for a given window, statically, for those situations where the window is only used to display output from some program, and not for interactive use. Both xterm and rxvt allow this, using the -n option to specify the icon name and the -T option to specify the title. You may also use X resources to specify icon name or title.

The Mac OS X Terminal application lets you set the title from the Set Title command on the Shell menu as well.

—DJF and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

X font names make the Rosetta Stone look like bedtime reading. Those hardy souls who want to experiment with fonts or access fonts on remote machines must take the high road and learn the X font naming conventions anyway. But if you just want to locate some fonts to use with xterm and other clients, you can use the predefined aliases for some of the constant-width fonts available on most systems.

Figure 5-2 lists the aliases for some constant-width fonts that should be appropriate for most of the standard clients, including xterm. [These "aliases" are basically font names. They aren't the same as shell aliases (Section 29.1). Also note that terminals should use constant-width fonts (where every character — thin or wide — occupies the same horizontal width). Constant-width fonts ensure that, for instance, the 54th character in every line of output from ls -l is always in the same horizontal position on the screen — so columns will always be straight. — JP] To give you an idea of the range of sizes, each alias is written in the font it identifies.

Figure 5-2: Miscellaneous fonts for xterm and other clients

In these cases, the aliases refer to the dimensions in pixels of each character in the font. (For example, "10×20" is the alias for a font with characters 10 pixels wide by 20 pixels high.) Note, however, that an alias can be virtually any character string.

The default font for many applications, including xterm, is a 6×13 pixel font that has two aliases: "fixed" and "6×13." Many users consider this font to be too small. If you have enough screen space, you might want to use the 10×20 font for xterm windows:

% xterm -fn 10x20 &

You can make this font the default for xterm by specifying it as the value for the font resource variable (Section 6.3):

XTerm*font: 10x20

Another quick way to get a list of fonts that match a given string is to use the xlsfonts program, which accepts a variety of options but may be used as simply as this:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

xterm has four different menus, each providing items that serve different purposes. You display a menu by placing the pointer on the window and simultaneously pressing the CTRL (keyboard) key and a pointer button. When you're using a window manager that provides a titlebar or frame, the pointer must rest within the window proper and not on any window decoration.

Table 5-3 describes the menus and how to display them.

Table 5-3: The xterm menus
Menu title	Display by holding	Use to
Main Options	CTRL, pointer button 1	Enter secure mode; interrupt, stop, etc., the xterm process.
VT Options	CTRL, pointer button 2	Toggle user preferences, including scrollbar, reverse video, margin bell; toggle Tektronix/VT100 mode.
VT Fonts	CTRL, pointer button 3	Select alternative display font.
Tek Options

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Ideally, you want to set up your environment so that xterm windows (and other clients) come up automatically with the characteristics you prefer, including the display font. I use the very large 10×20-pixel font (Section 5.16) for all my xterm windows by specifying the resource variable (Section 6.3):

XTerm*font: 10x20

But if you start an xterm and then decide you want a different font, you do have an option.

The xterm VT Fonts menu (Section 5.17) allows you to change a window's font on the fly, which is a very handy capability. You can change the font any number of times to accommodate a variety of uses. You might choose to use a large font for text editing; you could then change to a smaller font while a process is running, since you don't need to be reading or typing in that xterm. Since xterm's dimensions are determined by the number of characters wide by the number of lines high, changing the font also changes the size of the window.

When the focus is on an xterm, you display the menu by pressing CTRL and then the third pointer button. The default menu is shown in Figure 5-3.

Figure 5-3: xterm's VT Fonts menu lets you change fonts dynamically

The items on the VT Fonts menu are toggles, each of which provides a different size display font. If you have not toggled any items on this menu, a check mark will appear next to Default, which is the font specified when the xterm was run. This font could have been specified on the xterm command line or in a resource file. Whatever the case, this font remains the Default for the duration of the current xterm process.

By default, the Unreadable, Tiny, Small, Medium, Large, and Huge menu choices toggle the constant-width fonts shown in Table 5-4.

Table 5-4: VT Fonts menu defaults
Menu item	Default font

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The xclipboard client does exactly what you might think: it allows you to save multiple text selections (Section 5.13) and copy them to other windows. Text you copy from an xterm window can be made the CLIPBOARD selection (and thus automatically appear in the xclipboard window). To set this up, you first need to customize xterm using resources.

For text you copy from an xterm to be pasted automatically into xclipboard, the text must be made the CLIPBOARD selection. You set this up to happen by specifying a few translations (Section 6.4) for xterm. Here are the translations I use to coordinate xterm with xclipboard:

*VT100.Translations:    #override\

    Button1 <Btn3Down>: select-end(primary,CUT_BUFFER0,CLIPBOARD)\n\

    !Shift <Btn2Up>:    insert-selection(CLIPBOARD)\n\

    ~Shift ~Ctrl ~Meta <Btn2Up>:  insert-selection(primary,CUT_BUFFER0)

To let you store multiple text selections, the seemingly tiny xclipboard actually provides multiple screens, each of which can be thought of as a separate buffer. Each time you use the pointer to make text the CLIPBOARD selection, the xclipboard advances to a new screen in which it displays and stores the text. If you make a long selection, it might take up more than one screen, but the clipboard still considers it a single buffer. When you make a selection that extends beyond the bounds of the xclipboard window (either horizontally, vertically, or both), scrollbars (Section 5.11) will be activated in the window to allow you to view the entire selection.

To the right of the command buttons is a tiny box that displays a number corresponding to the selection currently in the xclipboard window. Once you have saved multiple selections, you can click on the client's Next and Prev command buttons to move forward and backward among these screens of text.

If you've coordinated xterm with xclipboard using the guidelines outlined earlier, you paste the CLIPBOARD selection in an xterm window by holding down the Shift key and clicking the second pointer button. When you paste the CLIPBOARD selection, you get the selection that's currently being displayed in the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you experiment making large selections with xclipboard, you may discover what seems to be a bug in the program. Though making a new selection usually causes the screen to advance and display the new text, this does not happen reliably after a selection that vertically spans more than one screenful. In these cases, the new selection is saved in the xclipboard (and the number in the small box is incremented to indicate this); however, the xclipboard window does not automatically advance to show you the new current selection. Instead, the previous long selection is still displayed. (For example, though the box says "5," indicating that a fifth selection has been saved, the window is still displaying selection #4.) This is a bit of xclipboard sleight of hand: the new selection has been successfully made, but the appearance of the window belies this fact. The Next button will probably add to your confusion; it will not be available for selection, suggesting that the text in the window is the last selection saved. This is not the case.

To get around this problem and display the actual current selection, press the Previous button. The same long selection (which is, in actuality, the Previous selection) will be displayed again. (The small box will flip back to display the preceding number as well.) Then the Next button will be enabled, and you can click on it to display the actual current selection. The selection displayed in the window and the number in the small box will correspond.

—VQ and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of my favorite uses for an xterm (which may seem natural to people who've grown up using window systems, but was a pleasant surprise for a guy who started computing with teletypes in 1970) is using a window to accept text pasted from some other window. For instance, in writing this book, I'll have one window open with something happening that I want to put into the book. So I select the text, then paste it into another xterm window — where there's usually a text editor (like vi, with its keymaps for pasting text (Section 18.5)).

You can also use a text editor or Unix utilities to reformat text from one window before pasting it into another. For instance, you'd like to send most of the text in your browser to another window where you're composing an email message. But the web site used those irritating Microsoft Windows-specific quote characters that show up as question marks (?) on any other platform. So you paste the text into an Emacs window, do a quick run of text substitution, and copy the result to paste into the email window.

Go to http://examples.oreilly.com/upt3 for more information on: requote

Another problem with email messages comes when you're sending a reply to someone who's used very long or jagged lines and the quoted text is a mess. But if you cut the messy text into an xterm window running the requote shell script, you'll get a neatened version. In the following example, the text I paste (cut from a quoted email message) is shown in boldface. Then I press CTRL-d, and the result appears; I can paste it back into the email message:

$ requote

               > This is a long line of text that runs on and on and wraps to the next

               line without a quote character at the start and it goes on and on and on

               and well you know

               > This is the next line of text

CTRL-d

> This is a long line of text that runs on and on and wraps to the next

> line without a quote character at the start and it goes on and on and

> on and well you know This is the next line of text

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The -e option to xterm is useful for running a single command before exiting. For example, if you just want to run a character-based mail program, type the following:

% xterm -e mail

When you quit the mail program, the xterm window exits.

The -e option needs to be the last xterm option on the command line. The remainder of the command line is assumed to be part of the command to be executed by xterm. The new window has the command name in its titlebar by default (unless overridden by other command-line options (Section 5.15)).

One use for xterm -e is for running a window with a login session to a remote system, like this:

% xterm -e ssh 

               hostname

                &

Go to http://examples.oreilly.com/upt3 for more information on: ssh-agent

The xterm process runs on the local system, but immediately logs you into the remote machine. You are prompted for a password in the new xterm that pops up — before you can log in to the remote system. This isn't as convenient as putting that command in your X setup file (like .xinitrc or .xsession) — but it's far more secure because you don't need to put your hostname in your .rhosts or .shosts file (Section 1.21), which is a potential security hole. (Or, if you use ssh for your remote login — and you start ssh-agent before you start X — you won't need to type passwords at all during your X session. This is the handiest setup by far.)

You can use -e to create a makeshift X display for any character-based programs you like to run. For example, you might want to keep track of messages sent to the console, but you can't run xterm -C to get console messages because you aren't actually logged in on the console. You might run something like this:

tail -f Section 12.10

% xterm -e tail -f /var/log/messages &

Section 24.21 has more about how this works.

—LM, JP, and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Being a belt-and-suspenders kind of guy, I've gotten in the habit of quoting arguments to commands. This makes good sense with lots of Unix shell commands, but it can get you in trouble with xterm -e. For example, I wanted to set up a job that would open vi in a window to edit a file named .postit. At first, I used the command:


               xterm ... -e 'vi .postit' &

only to receive the perplexing message in the resulting window:

Can't execvp vi .postit

The quotes passed the entire string to xterm as an argument, which parsed it as a single command name, rather than a command plus argument. Removing the quotes solved the problem.

—TOR and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you have a Linux system, you may want to use loadkeys instead of xmodmap. loadkeys is designed to set the keymap used by the system as a whole, particularly the console, so use your own judgment. Whatever is done in xmodmap will affect X but not the system console.

An important piece to the X Window System puzzle is filled by the xmodmap client. When the user performs any action — such as typing a key or moving the mouse — the server sends a packet of information to the client called an event. These events are then translated into actions by the client. You can use the xmodmap utility to effectively change the event that is reported to the client.

Keysym mappings are mappings of keyboard events at the server level, before the event is sent to the client. Keysyms are the symbols used for each key on the keyboard.

The X server maintains a keymap table , which contains a listing of keys on the keyboard and how they should be interpreted. A client gets the keymap table from the server upon client startup. In most cases, the keymap table is used to interpret keys literally — when you press the letter "a," a key code is sent to the client that corresponds to the letter "a" in the keymap table.

You can use the xmodmap client to reassign key codes within the keymap table. xmodmap can therefore be used to redefine how the key is interpreted by the client. You probably wouldn't want to translate the alphanumeric keys on the keyboard, but you may want to translate others. For example, you might want to change the BACKSPACE key to DELETE:

% xmodmap -e "keysym BackSpace = Delete"

Another example is if you mistakenly hit the CAPS LOCK key a bit too often, you can disable it completely. Some people might disable CAPS LOCK the low-tech way (by just removing the key from the keyboard!), but you can also render it harmless with the command:

% xmodmap -e "keysym Caps_Lock = "

effectively disabling the CAPS LOCK key entirely. Note that the symbol is now gone and can't be redefined without using the hardware key code.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you have a Linux system, you may want to use loadkeys instead of xmodmap. loadkeys is designed to set the keymap used by the system as a whole, particularly the console, so use your own judgment. Whatever is done in xmodmap will affect X but not the system console.

An important piece to the X Window System puzzle is filled by the xmodmap client. When the user performs any action — such as typing a key or moving the mouse — the server sends a packet of information to the client called an event. These events are then translated into actions by the client. You can use the xmodmap utility to effectively change the event that is reported to the client.

Keysym mappings are mappings of keyboard events at the server level, before the event is sent to the client. Keysyms are the symbols used for each key on the keyboard.

The X server maintains a keymap table , which contains a listing of keys on the keyboard and how they should be interpreted. A client gets the keymap table from the server upon client startup. In most cases, the keymap table is used to interpret keys literally — when you press the letter "a," a key code is sent to the client that corresponds to the letter "a" in the keymap table.

You can use the xmodmap client to reassign key codes within the keymap table. xmodmap can therefore be used to redefine how the key is interpreted by the client. You probably wouldn't want to translate the alphanumeric keys on the keyboard, but you may want to translate others. For example, you might want to change the BACKSPACE key to DELETE:

% xmodmap -e "keysym BackSpace = Delete"

Another example is if you mistakenly hit the CAPS LOCK key a bit too often, you can disable it completely. Some people might disable CAPS LOCK the low-tech way (by just removing the key from the keyboard!), but you can also render it harmless with the command:

% xmodmap -e "keysym Caps_Lock = "

effectively disabling the CAPS LOCK key entirely. Note that the symbol is now gone and can't be redefined without using the hardware key code.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The xev client is essential for debugging X Window System keysym mappings (Section 6.1). When you start up xev, a small "event window" appears. All events that take place within that window are shown on standard output. This means screenfuls of output, but it also means that when you type a key, you can immediately trace the resulting event. For example, if you need to know what keysym is sent when you type the DELETE key on the keyboard, just run xev and type the DELETE key in the event window. Typical output might be the following:

KeyPress event, serial 13, synthetic NO, window 0x800001,

   root 0x8006d, subw 0x800002, time 1762968270, (50,36),

   root:(190,176), state 0x0, keycode 27 (keysym 0xffff, Delete),

   same_screen YES, XLookupString gives 1 characters: "^?"



KeyRelease event, serial 15, synthetic NO, window 0x800001,

   root 0x8006d, subw 0x800002, time 1762968336, (50,36),

   root:(190,176), state 0x0, keycode 27 (keysym 0xffff, Delete),

   same_screen YES, XLookupString gives 1 characters: "^?"

This tells you that the DELETE key (keycode 27) is interpreted as keysym 0xffff, which is Delete and character ^?. If you do an

xmodmap

-pk

(Section 5.1), you should see a line resembling:

27      0xffff (Delete)

If you redefine the DELETE key as the BACKSPACE key and do the same exercise (run xev and press the DELETE key), you should see something like this:

% xmodmap -e "keysym Delete = BackSpace"

% xev

        ...

KeyPress event, serial 13, synthetic NO, window 0x800001,

   root 0x8006d, subw 0x800002, time 1763440073, (44,39),

   root:(240,235), state 0x0, keycode 27 (keysym 0xff08, BackSpace),

   same_screen YES, XLookupString gives 1 characters: "^H"



KeyRelease event, serial 15, synthetic NO, window 0x800001,

   root 0x8006d, subw 0x800002, time 1763440139, (44,39),

   root:(240,235), state 0x0, keycode 27 (keysym 0xff08, BackSpace),

   same_screen YES, XLookupString gives 1 characters: "^H"

This tells you that now the DELETE key (still keycode 27) is being interpreted as hexadecimal

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Virtually all X Window System clients are customizable. You can specify how a client looks on the screen — its size and placement, its border and background color or pattern, whether the window has a scrollbar, and so on. This article introduces X resources and shows their syntax.

Traditional Unix applications rely on command-line options to allow users to customize the way they work. X applications support command-line options too, but often not for all features. Almost every feature of an X program can be controlled by a variable called a resource; you can change the behavior or appearance of a program by changing the value associated with a resource variable.

Resource variables may be Boolean (such as

scrollBar:

True

) or take a numeric or string value (borderWidth: 2 or

foreground:

blue

). What's more, in applications written with the X Toolkit (or an Xt-based toolkit such as the Motif toolkit), resources may be associated with separate objects (or "widgets") within an application. There is a syntax that allows for separate control over both a class of objects in the application and an individual instance of an object. This is illustrated by these resource specifications for a hypothetical application called xclient:

xclient*Buttons.foreground:     blue

xclient*help.foreground:        red

The first resource specification makes the foreground color blue for all buttons in the xclient application (in the class Buttons); the second resource specification makes the foreground color red for the help button in this application (an instance of the class Buttons). Resource settings can be even simpler than this.

The values of resources can be set as application defaults using a number of different mechanisms, including resource files in your home directory and a program called xrdb (X resource database manager). As we'll see, the xrdb program stores resources directly in the X server, making them available to all clients, regardless of the machine on which the clients run.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This article introduces event translations, which are special X Window System resources that control actions of things like mouse clicks. Section 6.3 introduces X resources and shows their syntax. Section 6.5 through Section 6.9 explain how to set and check resources — as you log in and after.

We've discussed the basics of resource-naming syntax. From the sample resource settings, it appears that what many resource variables do is self-evident or nearly so. Among the less obvious resource variables, there is one type of specification, an event translation, that can be used with many clients and warrants somewhat closer examination.

User input and several other types of information pass from the server to a client in the form of events . An event is a packet of information that gives the client something to act on, such as keyboard input. Moving the pointer or pressing a key causes input events to occur. When a program receives a meaningful event, it responds with some sort of action.

For many clients, the resource manager recognizes mappings between certain input events (such as a pointer button click) and some sort of action by the client program (such as selecting text). A mapping between one or more events and an action is called a translation. A resource containing a list of translations is called a translation table.

Many event translations are programmed into an application and are invisible to the user. For our purposes we are only concerned with very visible translations of certain input events, primarily the translation of keystrokes and pointer button clicks to particular actions by a client program.

The operation of many clients, notably xterm, is partly determined by default input event translations. For example, selecting text with the first pointer button (an event) saves that text into memory (an action).

In this case, the input "event" is actually three separate X events:

Pressing the first pointer button.
Moving the pointer while holding down the first button.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Learning to write resource specifications is a fairly manageable task, once you understand the basic rules of syntax and precedence. In contrast, the multiple ways you can set resources — for a single system, multiple systems, a single user, or for all users — can be confusing. For our purposes, we are primarily concerned with specifying resources for a single user running applications both on the local system and on remote systems in a network.

As we've said, resources are generally specified in files. A resource file can have any name you like. Resources are generally "loaded" into the X server by the xrdb (Section 56.8) client, which is normally run from your startup file or run automatically by xdm when you log in. Prior to Release 2 of X, there was only one resource file called .Xdefaults, placed in the user's home directory. If no resource file is loaded into the server by xrdb, the .Xdefaults file will still be read.

Remember that X allows clients to run on different machines across a network, not just on the machine that supports the X server. One problem with the older .Xdefaults mechanism was that users who were running clients on multiple machines had to maintain multiple .Xdefaults files, one on each machine. By contrast, xrdb stores the application resources directly in the server, thus making them available to all clients, regardless of the machine on which the clients are running. As we'll see, xrdb also allows you to change resources without editing files.

Of course, you may want certain resources to be set on all machines and others to be set only on particular machines. For a complex setup, check the detailed information in O'Reilly & Associates' X Window System Guide, Volume 3M, Chapter 11.

In addition to loading resource files, you can specify defaults for a particular instance of an application from the command line using two options: -xrm and -name.

A sample resources file follows. This file sets the border width for all clients to a default value of two pixels, and it sets other specific variables for

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The -xrm command-line option, which is supported by all X Window System clients written with the X Toolkit, can be useful in specifying from the command line any specification that you would otherwise put into a resources file (Section 6.5). For example:

% xterm -xrm 'xterm*Foreground: blue' &

Note that a resource specification on the command line must be quoted using the single quotes.

The -xrm option only specifies the resource(s) for the current instance of the application. Resources specified in this way do not become part of the resource database.

The -xrm option is most useful for setting classes, since most clients have command-line options that correspond to instance variable names. For example, the -fg command-line option sets the foreground attribute of a window, but -xrm must be used to set Foreground.

Note also that a resource specified with the -xrm option will not take effect if a resource that takes precedence has already been loaded with xrdb. For example, say you've loaded a resource file that includes the specification:

xterm*pointerShape:  pirate

The command-line specification of another cursor will fail:

% xterm -xrm '*pointerShape:  gumby' &

because the resource xterm*pointerShape is more specific than the resource *pointerShape. Instead, you'll get an xterm with the previously specified pirate cursor.

To override the resource database (and get the Gumby cursor), you'd need to use a resource equally (or more) specific, such as the following:

% xterm -xrm 'xterm*pointerShape:  gumby' &

—VQ and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The command-line option -name lets you name one instance of an application; the server identifies the single instance of the application by this name. The name of an application affects how resources are interpreted. This option is supported by all X Window System clients written with the X Toolkit.

For example, the following command sets the xterm instance name to bigxterm:

% xterm -name bigxterm &

When this command is run, the client uses any resources specified for bigxterm rather than for xterm.

The -name option allows you to create different instances of the same application, each using different resources. For example, you could put the following entries into a resource file such as .Xresources:

XTerm*Font:          8x13

smallxterm*Font:     6x10

smallxterm*Geometry: 80x10

bigxterm*Font:       9x15

bigxterm*Geometry:   80x55

You could then use these commands to create xterms of different specifications. The command:

% xterm &

would create an xterm with the default specifications, while:

% xterm -name bigxterm &

would create a big xterm, 80 characters across by 55 lines down, displaying in the font 9x15. The command:

% xterm -name smallxterm &

would create a small xterm, 80 characters across by 10 lines down, displaying in the font 6x10.

—VQ and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The xrdb program saves you from maintaining multiple resource files if you run clients on multiple machines. It stores resources on the X server, where they are accessible to all clients using that server. (This property is also called the resource database.)

Place the appropriate xrdb command line in your .xinitrc file or .xsession file to initialize resources at login, although it can also be invoked interactively. It has the following syntax:


               xrdb [options] [filename]

The xrdb client takes several options, all of which are documented on its manual page. We'll discuss the most useful options.

The optional filename argument specifies the name of a file from which the values of client variables (resources) will be read. If no filename is specified, xrdb will expect to read its data from standard input. Note that whatever you type will override the previous contents, so if you inadvertently type xrdb without a filename argument and then quit with CTRL-d, you will delete any previous values. (You can append new settings to current ones using the -merge option discussed later in this article.)

The resource filename can be anything you want. Two commonly used names are .Xresources and .Xdefaults.

You should load a resource file with the xrdb -load option. For example, to load the contents of your .Xresources file into the RESOURCE_MANAGER, you would type:

% xrdb -load .Xresources

Querying the resource database

You can find out what options are currently set by using the -query option. For example:

% xrdb -query

XTerm*ScrollBar:     True

bigxterm*font:       9x15

bigxterm*Geometry:   80x55

smallxterm*Font:     6x10

smallxterm*Geometry: 80x10

xterm*borderWidth:   3

If xrdb has not been run, this command will produce no output.

Loading new values into the resource database

By default, xrdb reads its input (either a file or standard input) and stores the results into the resource database, replacing the previous values. If you simply want to merge new values with the currently active ones (perhaps by specifying a single value from standard input), you can use the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The appres (application resource) program lists the resources that currently might apply to a client. These resources may be derived from several sources, including the user's .Xresources file and a system-wide application defaults file. The directory /usr/lib/X11/app-defaults contains application-default files for several clients. (Note that it may be in a different place depending on how your X11 is installed; on Mac OS X, which does not come with X by default, you might find it in /usr/X11R6/etc/app-defaults in one popular install or /usr/local/lib/X11/app-defaults in another.) The function of these files is discussed in the next section. For now, be aware that all of the resources contained in these files begin with the class name of the application.

Also be aware that appres has one serious limitation: it cannot distinguish between valid and invalid resource specifications. It lists all resources that might apply to a client, regardless of whether the resources are correctly specified.

appres lists the resources that apply to a client having the class_name and/or instance_name you specify. Typically, you would use appres before running a client program to find out what resources the client program will access.

For example, say you want to run xterm, but you can't remember the latest resources you've specified for it, whether you've loaded them, what some of the application defaults are, etc. You can use the appres client to check the current xterm resources. If you specify only a class name, as in this command line:

% appres XTerm

appres lists the resources that any xterm would load. In the case of xterm, this is an extensive list, encompassing all of the system-wide application defaults, as well as any other defaults you have specified in a resource file.

You can also specify an instance name to list the resources that applies to a particular instance of the client, as in:

% appres XTerm bigxterm

If you omit the class name, xappres

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the unique advantages of window systems such as X is that you can run applications remotely and view them on the local display (as opposed to systems that merely allow for the execution of shared applications by the local host, such as Windows and the Mac OS prior to OS X). Even Mac OS X, except insofar as it can run an X server, does not allow for a split between an application's display and its execution. Only X-aware applications may be executed in such a fashion.

You can try this easily enough by doing an rlogin or telnet to the remote host, setting the DISPLAY environment variable and starting up an X client. Of course, it helps to have an X server already running on your local machine. In the following example, we start up a new xload client running on the host ruby:

sapphire:joan % rlogin ruby

Password:

Last login: Mon Mar 12 16:27:23 from sapphire.oreilly.com

NetBSD 1.4.2A (ORA-GENERIC) #6: Wed May 31 06:12:46 EEST 2000



TERM = (vt100) xterm



ruby:joan % setenv DISPLAY sapphire:0

ruby:joan % xload &

(You must, of course, have an account on the remote system.)

The first thing that might go wrong is that you may run into server access control. If you see the following error:

Xlib:  connection to "sapphire:0" refused by server

Xlib:  Client is not authorized to connect to Server

Error: Can't open display: sapphire:0

you can probably fix it by typing xhost +ruby in a sapphire window and running the command again on ruby.

Once you have networking and access control issues solved, you should be able to display clients from the remote machine. The next issue is how to run remote clients easily.

If you have ssh ( Section 1.21), its X forwarding handles authorization (setting DISPLAY) and also encrypts the connection to make it secure. Here's an example using ssh for an interactive login:

sapphire:joan % ssh ruby

joan's passphrase:

Last login: Mon Mar 12 16:27:23 from sapphire.oreilly.com

NetBSD 1.4.2A (ORA-GENERIC) #6: Wed May 31 06:12:46 EEST 2000



TERM = (vt100)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Computers and offices have one thing in common: you lose things in them. If you walk into my office, you'll see stacks of paper on top of other stacks of paper, with a few magazines and business cards in the mix. I can often find things, but I'd be lying if I said that I could always find that article I was reading the other day!

When you look at a new computer user's home directory (Section 31.11) , you often see something similar to my office. You see a huge number of unrelated files with obscure names. He hasn't created any subdirectories, aside from those the system administrator told him they needed; and those probably aren't even being used. His home directory probably contains programs for several different projects, personal mail, notes from meetings, a few data files, some half-finished documentation, a spreadsheet for something he started last month but has now forgotten, and so on.

Remember that a computer's filesystem isn't that much different from any other filing system. If you threw all of your papers into one giant filing cabinet without sorting them into different topics and subtopics, the filing cabinet wouldn't do you much good at all: it would just be a mess. On a computer, the solution to this problem is to sort your files into directories, which are analogous to the filing cabinets and drawers.

The Unix filesystem can help you keep all of your material neatly sorted. Your directories are like filing cabinets, with dividers and folders inside them. In this chapter, we'll give some hints for organizing your computer "office." Of course, things occasionally get misplaced even in the most efficient offices. Later we'll show some scripts that use the find (Section 8.3) and grep (Section 9.21) commands to help you find files that are misplaced.

— ML

Various operating systems store users' home directories in many places, and you've probably already noticed evidence of this throughout this book. Home directories may be in /home/ username, /u/ username

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Computers and offices have one thing in common: you lose things in them. If you walk into my office, you'll see stacks of paper on top of other stacks of paper, with a few magazines and business cards in the mix. I can often find things, but I'd be lying if I said that I could always find that article I was reading the other day!

When you look at a new computer user's home directory (Section 31.11) , you often see something similar to my office. You see a huge number of unrelated files with obscure names. He hasn't created any subdirectories, aside from those the system administrator told him they needed; and those probably aren't even being used. His home directory probably contains programs for several different projects, personal mail, notes from meetings, a few data files, some half-finished documentation, a spreadsheet for something he started last month but has now forgotten, and so on.

Remember that a computer's filesystem isn't that much different from any other filing system. If you threw all of your papers into one giant filing cabinet without sorting them into different topics and subtopics, the filing cabinet wouldn't do you much good at all: it would just be a mess. On a computer, the solution to this problem is to sort your files into directories, which are analogous to the filing cabinets and drawers.

The Unix filesystem can help you keep all of your material neatly sorted. Your directories are like filing cabinets, with dividers and folders inside them. In this chapter, we'll give some hints for organizing your computer "office." Of course, things occasionally get misplaced even in the most efficient offices. Later we'll show some scripts that use the find (Section 8.3) and grep (Section 9.21) commands to help you find files that are misplaced.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Various operating systems store users' home directories in many places, and you've probably already noticed evidence of this throughout this book. Home directories may be in /home/ username, /u/ username, /Users/ username, or some other, more esoteric location.

The simplest way to find out where your system believes your home directory to be is to take advantage of the fact that cd with no arguments changes to your home directory:

% cd

% pwd

/home/users/deb

Generally, the $HOME environment variable will point to your home directory:

% echo $HOME

/home/users/deb

Most shells also expand tilde (~ ) to a user's home directory as well, so ~/archive on my machine becomes /home/users/deb/archive and ~joel/tmp expands to /home/users/joel/tmp.

Your home directory is set in your /etc/passwd entry (or equivalent — Netinfo on Darwin and NIS on Solaris store the same information, for example). There is no actual requirement that all users' home directories be in the same directory. In fact, I've seen systems that have lots of users organize home directories by the first few letters of the username (so my home directory there was /home/d/de/deb).

If you add user accounts using a tool rather than by using vipw and adding them by hand, take a peek at the documentation for your tool. It should tell you both where it wants to put home directories by default and how to change that default should you want to.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix uses the same mode bits ( Section 50.2) for directories as for files, but they are interpreted differently. This interpretation will make sense if you remember that a directory is nothing more than a list of files. Creating a file, renaming a file, or deleting a file from a directory requires changing this list: therefore, you need write access to the directory to create or delete a file. Modifying a file's contents does not require you to change the directory; therefore, you can modify files even if you don't have write access to the directory (provided that you have write access to the file).

Reading a directory is relatively straightforward: you need read access to list the contents of a directory (find out what files it contains, etc.). If you don't have read access, you can't list the contents of the directory. However (surprise!), you may still be able to access files in the directory, provided that you already know their names.

Execute access for a directory has no meaning per se, so the designers of Unix have reassigned this. It is called the search bit. Search access is needed to perform any operation within a directory and its subdirectories. In other words, if you deny execute access to a directory, you are effectively denying access to the directory and everything beneath it in the directory tree. Note that providing search access to a directory without read access prevents people from listing the directory, but allows them to access files if they know their names. This is particularly useful in situations where you want to allow public access to areas, but only to people who know exactly what files to access; files available via a web server are a good example.

The SUID bit (Section 50.4) is meaningless for directories, but the SGID bit set on a directory affects group ownership of files created in that directory, and the sticky bit prohibits users with write access to the directory from deleting or renaming files that they don't own.

The exception is, of course, that the superuser can do absolutely anything at any time.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you compile programs or write shell scripts, it's good to put them in one directory. This can be a subdirectory of your home directory. Or, if several people want to use these programs, you could pick any other directory — as long as you have write access to it. Usually, the directory's name is something like bin — though I name mine .bin (with a leading dot) to keep it from cluttering my ls listings.

For instance, to make a bin under your home directory, type:

% cd

% mkdir bin

Once you have a directory for storing programs, be sure that the shell can find the programs in it. Type the command echo $PATH and look for the directory's pathname. For instance, if your directory is called /u/walt/bin, you should see:

% echo $PATH

...:/u/walt/bin:...

If the directory isn't in your PATH, add it in your .profile or .cshrc.

If other people are using your bin directory, use a command like chmod go+rx bin to give them access. If you're concerned about security, prevent unauthorized users from adding, removing, or renaming files in your directory by making sure that only you have write access; you can do this with a command like chmod go-w bin . Also be sure that individual files can't be edited by people who shouldn't have access to the files.

When you add a new program to your bindirectory, if you use the C shell or a C-shell derivative, you need to use the shell's rehash command to update its command search path.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You might want to create a private directory for your personal files: love letters, financial data, complaints about your boss, off-color jokes, or whatever you want to keep there. While you can set any directory you own to be private, having one in your home directory is convenient to organize all of your private directories together. For simplicity, you can just name it private; giving it a less obvious name, however, can make it more difficult for prying eyes to discover.

Once you've created a private directory, you should set its file access mode ( Section 50.2) to 700; this means that you're the only person allowed to read, write, or even list the files that are in the directory. Here's how:

% mkdir private

% chmod 700 private

On any Unix system, anyone who knows the root password can become superuser ( Section 49.9) and read any files he wants. So a private personal directory doesn't give you complete protection by any means — especially on systems where most users know the root password. If you really need security, you can always encrypt your files.

—ML and DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Let's think about a filing cabinet again. If the files in your filing cabinet were called letter1, letter2, letter3, and so on, you'd never be able to find anything — the names aren't descriptive enough. The same is true on your computer — you should come up with a descriptive name for each file that you create. Unix systems let you have very long filenames. A few older systems have a 14-character limit, but most allow names that are 256 characters long — hopefully, longer than you will ever need.

Generally, a descriptive filename summarizes the contents with a few useful words. letter is not a terribly useful summary, unless perhaps you've only ever written one letter and don't expect to write another. The recipient's name (JohnShmoe, for example) would only be a useful summary if you expect to send only one letter to that person. Even if you only plan to send one letter, the name doesn't tell you anything about what you sent Mr. Shmoe.

OctoberGoldPriceTrends is a pretty good summary; it's obvious what the contents of that file are, though you might want to know to which year it referred, looking back two years from now. I often start time-specific files with the date, so that ls sorts the files in date order. If you do this, I recommend a YYYYMMDD format to get proper sorting, so files look like 20021004-GoldPrices. If you're going to have regular updates to something, you might want to make a directory to hold those things (e.g., GoldPrices/20021004, GoldPrices/20021108, GoldPrices/20021206, and so forth). Note that in this specific example, a filename of nothing but a date makes sense, because you don't have anything else in that directory but information on gold prices.

Bruce Barnett has suggested that, by using long filenames, you can create a simple "relational database." For example, you could find out everything you've recorded about the price of gold with a command like more *Gold*Price*. Of course, if this starts to get very complex, using an actual database is much simpler.

Similarly, if you're a programmer, the name of each file in your program should describe what the code does. If the code diagonalizes matrices, the file should be called something like

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Creating many directories has several advantages:

First, it is easier to find any particular file if your home directory is well sorted. Imagine a rack of filing cabinets that isn't sorted; people just insert files wherever they fit. You may as well throw your data out; when you need something, you'll never be able to find it.
Second, Unix can access files much faster when directories are relatively small. Ideally, directories should have at most 60 files in them.
Third, directories are an important part of Unix file protections. By setting the permissions on the directories themselves, you can use directories to help protect certain groups of files against access by others.

Create new directories liberally! Make a new directory for every new project you start; make subdirectories within these directories for subtopics. Your home directory should ideally contain nothing but subdirectories. Following are some recommended conventions.

If you're a programmer, create a new directory for each project. In the project directory, create a directory called src for source files, a directory called doc or man for documentation, a directory called obj for object files, a directory called rel for the current working version (or almost-working version) of the program, a directory called test for test files and results, and so on. If the program is large, your src and obj directories should also be split into different subdirectories, each containing different parts of the project (or perhaps the subdirectory for each part of the project should have its own src and obj directories).

Many users save all of their mail in one directory (often called Mail or Maildir, depending on your mail system), which is then divided into subdirectories by topic. I use a variation of this scheme; I keep general mail in my Mail directory, but I save correspondence about particular projects with the project itself. For example, my Power Tools mail is shelved with the source code for this article.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Earlier we told you that you should have lots of directories. Experienced Unix users are creating new directories all the time. How do you make a directory?

It's easy. Use the mkdir command, followed by the name of your new directory:

% mkdir 

               directory

This creates the new directory you want. It doesn't necessarily have to be in your current directory. For example:

% cd /home/los/mikel

% mkdir /src/books/power/articles/files

The only requirements are:

The parent of the directory you want to create must exist (in this case, /src/books/power/articles).
You must have write access to the parent directory.

What if the parent directory doesn't already exist? Assume, for example, that /src/books already exists, but the power and articles directories do not. You can make these "by hand," or on many Unix systems you can add the -p (parents) option:

% mkdir -p /src/books/power/articles/files

This tells mkdir to create all the intermediate directories that are needed. So the previous command creates three directories:

/src/books/power

/src/books/power/articles

/src/books/power/articles/files

If your mkdir doesn't have -p, you can use history substitution :

% mkdir /src/books/power

% !!/articles

mkdir /src/books/power/articles

% !!/files

mkdir /src/books/power/articles/files

On some mkdirs, you can also supply the file protection mode to be assigned to the directory. (By default, the file protection mode is derived from your umask.) To do so, use the -m option. For example:

% mkdir -m 755 /src/books/power/articles/files

This creates the directory with access mode 755, which allows the owner to do anything with the directory. Note that this must be a numeric mode.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A computer isn't that much different from a house or an office; unless you're incredibly orderly, you spend a lot of time looking for things that you've misplaced. Even if you are incredibly orderly, you still spend some time looking for things you need — you just have a better idea of where to find them. After all, librarians don't memorize the location of every book in the stacks, but they do know how to find any book, quickly and efficiently, using whatever tools are available. A key to becoming a proficient user of any system, then, is knowing how to find things.

This chapter is about how to find things. We're excluding the find (Section 9.1) utility itself because it's complicated and deserves a chapter of its own. We'll concentrate on simpler ways to find files, beginning with some different ways to use ls.

Well, okay, towards the end of the chapter we'll touch on a few simple uses of find, but to really get into find, take a peek at Chapter 9.

— ML

When you're talking to experienced Unix users, you often hear the terms " change time" and "modification time" thrown around casually. To most people (and most dictionaries), "change" and "modification" are the same thing. What's the difference here?

The difference between a change and a modification is the difference between altering the label on a package and altering its contents. If someone says chmod a-w myfile, that is a change; if someone says echo foo >> myfile, that is a modification. A change modifies the file's inode; a modification modifies the contents of the file itself. A file's modification time is also called the timestamp .

As long as we're talking about change times and modification times, we might as well mention "access times," too. The access time is the last time the file was read or written. So reading a file updates its access time, but not its change time (information about the file wasn't changed) or its modification time (the file itself wasn't changed).

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A computer isn't that much different from a house or an office; unless you're incredibly orderly, you spend a lot of time looking for things that you've misplaced. Even if you are incredibly orderly, you still spend some time looking for things you need — you just have a better idea of where to find them. After all, librarians don't memorize the location of every book in the stacks, but they do know how to find any book, quickly and efficiently, using whatever tools are available. A key to becoming a proficient user of any system, then, is knowing how to find things.

This chapter is about how to find things. We're excluding the find (Section 9.1) utility itself because it's complicated and deserves a chapter of its own. We'll concentrate on simpler ways to find files, beginning with some different ways to use ls.

Well, okay, towards the end of the chapter we'll touch on a few simple uses of find, but to really get into find, take a peek at Chapter 9.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you're talking to experienced Unix users, you often hear the terms " change time" and "modification time" thrown around casually. To most people (and most dictionaries), "change" and "modification" are the same thing. What's the difference here?

The difference between a change and a modification is the difference between altering the label on a package and altering its contents. If someone says chmod a-w myfile, that is a change; if someone says echo foo >> myfile, that is a modification. A change modifies the file's inode; a modification modifies the contents of the file itself. A file's modification time is also called the timestamp .

As long as we're talking about change times and modification times, we might as well mention "access times," too. The access time is the last time the file was read or written. So reading a file updates its access time, but not its change time (information about the file wasn't changed) or its modification time (the file itself wasn't changed).

Incidentally, the change time or "ctime" is incorrectly documented as the "creation time" in many places, including some Unix manuals. Do not believe them.

— CT

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Your directory might have 50, 100, or more files. Which files haven't been used for a while? You might save space by removing them. You read or edited a file yesterday, but you can't remember its name? These commands will help you find it. (If you want a quick review of Unix file times, see Section 8.2.)

In this example, I'll show you my bin (Section 7.4) directory full of shell scripts and other programs — I want to see which programs I don't use very often. You can use the same technique for directories with text or other files.

The ls command has options to change the way it orders files. By default, ls lists files alphabetically. For finding old files, use the -t option. This sorts files by their modification time, or the last time the file was changed. The newest files are listed first. Here's what happens:

jerry@ora ~/.bin

60 % ls -t

weather       unshar        scandrafts    rn2mh         recomp

crontab       zloop         tofrom        rmmer         mhprofile

rhyes         showpr        incc          mhadd         append

rhno          rfl           drmm          fixsubj       README

pickthis      maillog       reheader      distprompter  rtfm

cgrep         c-w           zrefile       xmhprint      saveart

dirtop        cw            zscan         replf         echoerr

which         cx            zfolders      fols

tcx           showmult      alifile       incs

I just added a shell script named weather yesterday; you can see it as the first file in the first column. I also made a change to my script named crontab last week; it's shown next. The oldest program in here is echoerr; it's listed last.

ls -t is also great for file-time comparisons in a script (Section 8.15). ls -t is quite useful when I've forgotten whether I've edited a file recently. If I've changed a file, it will be at or near the top of the ls -t listing. For example, I might ask, "Have I made the changes to that letter I was going to send?" If I haven't made the changes (but only think I have), my letter will most likely appear somewhere in the middle of the listing.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

By default, ls lists just one directory. If you name one or more directories on the command line, ls will list each one. The -R (uppercase R) option lists all subdirectories, recursively. That shows you the whole directory tree starting at the current directory (or the directories you name on the command line).

This list can get pretty long; you might want to pipe the output to a pager program such as less ( Section 12.3). The ls -C option is a good idea, too, to list the output in columns. (When the ls output goes to a pipe, many versions of ls won't make output in columns automatically.)

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you give ls the pathname of a directory, ls lists the entries in the directory:

% ls -l /home/joanne

total 554

-rw-r--r--  1 joanne      15329 Oct  5 14:33 catalog

-rw-------  1 joanne      58381 Oct 10 09:08 mail

   ...

With the -d option, ls lists the directory itself:

% ls -ld /home/joanne

drwxr-x--x  7 joanne       4608 Oct 10 10:13 /home/joanne

The -d option is especially handy when you're trying to list the names of some directories that match a wildcard. Compare the listing with and without the -d option:

% ls -Fd [a-c]*

arc/                    bm/                     ctrl/

atcat.c                 cdecl/

atl.c.Z                 cleanscript.c

% ls -F [a-c]*

atcat.c                 atl.c.Z                 cleanscript.c



arc:

BugsEtc.Z       arcadd.c        arcext.c.Z      arcmisc.c.Z

   ...

bm:

Execute.c.Z     MakeDesc.c.Z    MkDescVec.c.Z   Search.c.Z

   ...

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The GNU ls command — which is on a lot of systems, including Linux — can display names in colors. For instance, when I enable color listings on my system, directory names are in dark blue, symbolic links are in sky blue, executable files (scripts, programs, etc.) are in green, and so on.

tcsh 's built-in ls -F command can display in colors, too. Just set color in your .cshrc to enable it, and configure it using LS_COLORS as described later in this section. You may also want to look at Section 8.6.4 for another way to configure colors if - - color doesn't seem to work.

Go to http://examples.oreilly.com/upt3 for more information on: GNU ls

Has your system been set up for this? Simply try this command:

$ ls --color / /bin

If you don't get an error (ls: no such option — color, or something similar), you should see colors. If you don't get an error, but you also don't get colors, try one of these commands, and see what you get:

$ ls --color=always / /bin | cat -v

^[[00m/:

^[[01;34mbin^[[00m

^[[01;34mboot^[[00m

   ...

^[[01;34mvar^[[00m



/bin:

^[[01;32march^[[00m

^[[01;36mawk^[[00m

^[[01;32mbasename^[[00m

   ...



$ ls --color=yes / /bin | cat -v

                  ...same kind of output...

Those extra characters surrounding the filenames, such as ^[[01;34m and ^[[00m, are the escape sequences that (you hope) make the colors. (The cat -v ( Section 12.4) command makes the sequences visible, if there are any to see.) The ^[ is an ESC character; the next [ starts a formatting code; the 01 code means "boldface"; the semicolon (;) is a code separator; the 34 means "blue"; and the m ends the escape sequence. ^[[00m is an escape sequence that resets the attributes to normal. If you see the escape sequences when you use cat -v, but you haven't gotten any highlighting effects when you don't use it, there's probably some kind of mismatch between your

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A lot of the GNU utilities came from Unix utilities — but with extra features. The GNU ls command is no exception: as its info page (Section 2.9) says, "Because ls is such a fundamental program, it has accumulated many options over the years." Amen. Let's look at three of the options that aren't covered by other articles on ls.

An Emacs editor backup file (Section 19.4) has a name ending in ~ (tilde). If you use Emacs a lot, these files can really clutter your directories. The ls -B option ignores Emacs backup files:

$ ls

bar.c  bar.c~  baz.c  baz.c~  foo.c  foo.c~

$ ls -B

bar.c  baz.c  foo.c

The option -I (uppercase letter I) takes -B one step further: you can give a wildcard expression (shell wildcard pattern, not grep-like expressions) for entries not to list. (Remember that — because you want to pass the wildcard pattern to ls, and not let the shell expand it first — you need to quote (Section 27.12) the pattern.) For instance, to skip all filenames ending in .a and .o, use the wildcard pattern *.[ao], like this:

$ ls

bar.a  bar.c  bar.o  baz.a  baz.c  baz.o  foo.a  foo.c  foo.o

$ ls -I "*.[ao]"

bar.c  baz.c  foo.c

The "minimalist" side of me might argue that both -B and -I are feeping creatures because you can get basically the same effect by combining plain old ls with one of the "not this file" shell wildcard operators. This next option is in the same category. Instead of using -S to sort the files by size, you could pipe the output of plain ls -l to sort -n (Section 22.5) and sort on the size field, then strip off the information you didn't want and . . . ahem. (Grumble, grumble.) Okay, -S really is pretty useful. ;-) I use it a lot when I'm cleaning out directories and want to find the most effective files to remove:

$ ls -lS

total 1724

-rw-rw-r--    1 jerry    ora   395927 Sep  9 06:21 SunTran_map.pdf

-rw-------    1 jerry    ora   389120 Oct 31 09:55 core

-rw-r--r--    1 jerry    ora   178844 May  8 16:36 how

-rw-------    1 jerry    ora    77122 Oct 29 08:46 dead.letter

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Looking for a recently changed file? Not sure of the name? Trying to do this in a directory with lots of files? Try the lr alias:

alias lr "ls -lagFqt \!* | head"

This alias takes advantage of the -t option (Section 8.3) to ls, so that recent files can float to the top of the listing. !* is the csh syntax for "put all of the arguments to the alias here." (We have to escape the exclamation point to keep it from being interpreted when we set the alias.) head (Section 12.12) shows just the first ten lines.

A simple lr in my home directory gives me:

bermuda:home/dansmith :-) lr

total 1616

-rw-------  1 dansmith staff      445092 Oct  7 20:11 .mush256

-rw-r--r--  1 dansmith staff        1762 Oct  7 20:11 .history

drwxr-xr-x 30 dansmith staff        1024 Oct  7 12:59 text/

-rw-------  1 dansmith staff      201389 Oct  7 12:42 .record

drwxr-xr-x 31 dansmith staff        1024 Oct  4 09:41 src/

-rw-r--r--  1 dansmith staff        4284 Oct  4 09:02 .mushrc

   ...

You can also give a wildcarded pattern to narrow the search. For example, here's the command to show me the dot files that have changed lately:

bermuda:home/dansmith :-) lr .??*

-rw-------  1 dansmith staff      445092 Oct  7 20:11 .mush256

-rw-r--r--  1 dansmith staff        1762 Oct  7 20:11 .history

-rw-------  1 dansmith staff      201389 Oct  7 12:42 .record

-rw-r--r--  1 dansmith staff        4284 Oct  4 09:02 .mushrc

   ...

— DS

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The ls command normally ignores any files whose names begin with a dot (.). This is often very convenient: Unix has lots of small configuration files, scratch files, etc. that you really don't care about and don't want to be bothered about most of the time. However, there are some times when you care very much about these files. If you want to see "hidden" files, use the command ls -a. For example:

% cd

% ls                                  

               Don't show hidden files

Mail       mail.txt      performance   powertools

% ls -a                               

               This time, show me EVERYTHING

.        .emacs        Mail          powertools

..       .login        mail.txt

.cshrc   .mailrc       performance

With the -a option, we see four additional files: two C-shell initialization files, the customization files for the GNU Emacs editor, and mail. We also see two "special" entries, . and .., which represent the current directory and the parent of the current directory. All Unix directories contain these two entries (Section 10.2).

If you don't want to be bothered with . and .., many versions of ls also have a -A option:

% ls -A   

               Show me everything but . and ..

.cshrc   .login        Mail          performance

.emacs   .mailrc       mail.txt      powertools

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Because ls is one of the most commonly used Unix commands and provides numerous options, it's a good idea to create aliases for the display formats that best suit your needs. For example, many users always want to know about their "hidden" files. That's reasonable — they're just as important as any other files you have. In some cases, they can grow to take up lots of room (for example, some editors hide backup files), so it's worth being aware of them.

Rather than typing ls -a every time, you can create a convenient alias that supplies the -a or -A option ( Section 8.9) automatically:

$ alias la="ls -aF"

% alias la ls -aF

or:

$ alias la="ls -AF"

% alias la ls -AF

Two things to note here. First, I recommend using la as the name of the alias, rather than just renaming ls. I personally think it's dangerous to hide the pure, unadulterated command underneath an alias; it's better to pick a new name and get used to using that name. If you ever need the original ls for some reason, you'll be able to get at it without problems.

Second, what's with the -F option? I just threw it in to see if you were paying attention. It's actually quite useful; many users add it to their ls aliases. The -F option shows you the type of file in each directory by printing an extra character after each filename. Table 8-1 lists what the extra character can be.

Table 8-1: Filename types listed by ls -F
Character	Definition
(nothing)	The file is a regular file.
*	The file is an executable.
/

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

What's wrong here?

% ls

afile    exefiles   j       toobig

% lpr afile

lpr: afile: No such file or directory

Huh? ls shows that the file is there, doesn't it? Try using:

-v Section 12.4, -t -e Section 1125

% ls -l | cat -v -t -e

total 89$

-rw-rw-rw-  1 jerry          28 Mar  7 19:46 afile $

-rw-r--r--  1 root        25179 Mar  4 20:34 exefiles$

-rw-rw-rw-  1 jerry         794 Mar  7 14:23 j$

-rw-r--r--  1 root          100 Mar  5 18:24 toobig$

The cat -e option marks the ends of lines with a $. Notice that afile has a $ out past the end of the column. Aha . . . the filename ends with a space. Whitespace characters like TABs have the same problem, though the default ls -q (Section 8.12) option (on many Unix versions) shows them as ? if you're using a terminal.

If you have the GNU version of ls, try its -Q option to put double quotes around each name:

$ ls -Q

"afile "  "exefiles"  "j"  "toobig"

To rename afile, giving it a name without the space, type:

% mv "afile " afile

The quotes (Section 27.12) tell the shell to include the space as part of the first argument it passes to mv. The same quoting works for other Unix commands as well, such as rm.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

From time to time, you may get filenames with nonprinting characters, spaces, and other garbage in them. This is usually the result of some mistake — but it's a pain nevertheless.

If you're using a version of ls that uses -q by default (and most do these days), the ls command gives you some help; it converts all nonprinting characters to a question mark (?), giving you some idea that something funny is there. For example:

% ls

ab??cd

This shows that there are two nonprinting characters between ab and cd. To delete (or rename) this file, you can use a wildcard pattern like ab??cd.

Be careful: when I was new to Unix, I once accidentally generated a lot of weird filenames. ls told me that they all began with ????, so I naively typed

rm

????*

. That's when my troubles began. See Section 14.3 for the rest of the gruesome story. (I spent the next day and night trying to undo the damage.) The moral is: it's always a good idea to use echo to test filenames with wildcards in them.

If you're using an ls that came from System V Unix, you have a different set of problems. System V's ls doesn't convert the nonprinting characters to question marks. In fact, it doesn't do anything at all — it just spits these weird characters at your terminal, which can respond in any number of strange and hostile ways. Most of the nonprinting characters have special meanings — ranging from "don't take any more input" to "clear the screen." [If you don't have a System V ls, but you want this behavior for some reason, try GNU ls with its -N option. — JP]

To prevent this, or to see what's actually there instead of just the question marks, use the -b option. This tells ls to print the octal value of any nonprinting characters, preceeded by a backslash. For example:

% ls -b

ab\013\014cd

This shows that the nonprinting characters have octal values 13 and 14, respectively. If you look up these values in an ASCII table, you will see that they correspond to CTRL-k and CTRL-l. If you think about what's happening — you'll realize that CTRL-l is a formfeed character, which tells many terminals to clear the screen. That's why the regular

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I use awk (Section 20.10) a lot. One of my favorite features of awk is its associative arrays. This means awk can use anything as an index into an array. In the next example, I use the output of the file (Section 12.6) command as the index into an array to count how many files there are of each type:

xargs Section 28.17

#!/bin/sh

# usage: count_types [directory ...]

# Counts how many files there are of each type

# Original by Bruce Barnett

# Updated version by yu@math.duke.edu (Yunliang Yu)

find ${*-.} -type f -print | xargs file |

awk '{

        $1=NULL;

        t[$0]++;

}

END {

        for (i in t) printf("%d\t%s\n", t[i], i);

}' | sort -nr   # Sort the result numerically, in reverse

The output of this might look like:

38  ascii text

32  English text

20  c program text

17  sparc executable not stripped

12  compressed data block compressed 16 bits

8   executable shell script

1   sparc demand paged dynamically linked executable

1   executable /bin/make script

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you find a large directory and most of the files are new, that directory may not be suitable for removal, as it is still being used. Here is a script that lists a summary of file sizes, broken down into the time of last modification. You may remember that ls -l will list the month, day, hour, and minute if the file is less than six months old and show the month, day, and year if the file is more than six months old. Using this, the script creates a summary for each of the last six months, as well as a summary for each year for files older than that:

xargs Section 28.17

#!/bin/sh

# usage: age_files [directory ...]

# lists size of files by age

#

# pick which version of ls you use

#   System V

#LS="ls -ls"

#   Berkeley

LS="ls -lsg"

#

find ${*:-.} -type f -print | xargs $LS | awk  '

# argument 7 is the month; argument 9 is either hh:mm or yyyy

# test if argument is hh:mm or yyyy format

{

   if ($9 !~ /:/) {

      sz[$9]+=$1;

   } else {

      sz[$7]+=$1;

   }

}

END {

   for (i in sz) printf("%d\t%s\n", sz[i], i);

}' | sort -nr

The program might generate results like this:

5715   1991

3434   1992

2929   1989

1738   Dec

1495   1990

1227   Jan

1119   Nov

953   Oct

61   Aug

40   Sep

[For the book's third edition, I thought about replacing this venerable ten-year-old script with one written in Perl. Perl, after all, lets you get at a file's inode information directly from the script, without the ls -awk kludge. But I changed my mind because this technique — groveling through the output of ls -l with a "summarizing" filter script — is really handy sometimes. — JP]

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's a quick alias that figures out which file in a group is the newest:

-d Section 8.5

alias newer "ls -dt \!* | head -1"

If your system doesn't have a head ( Section 12.12) command, use sed 1q instead.

For example, let's say that you have two files named plan.v1 and plan.v2. If you're like me, you (often) edit the wrong version by mistake — and then, a few hours later, can't remember what you did. You can use this alias to figure out which file you changed most recently:

% newer plan.v*

plan.v1

I could also have used command substitution (Section 28.14) to handle this in one step:

% emacs `newer plan.*`

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One problem with symbolic links is that they're relatively "fragile" (Section 10.6). The link and the file itself are different kinds of entities; the link only stores the name of the "real" file. Therefore, if you delete or rename the real file, you can be left with a "dead" or "old" link: a link that points to a file that doesn't exist.

This causes no end of confusion, particularly for new users. For example, you'll see things like this:

% ls -l nolink

lrwxrwxrwx   1 mikel     users    12 Nov  2 13:57 nolink -> /u/joe/afile

% cat nolink

cat: nolink: No such file or directory

The file's obviously there, but cat tells you that it doesn't exist.

There's no real solution to this problem, except to be careful. Try writing a script that checks links to see whether they exist. Here's one such script from Tom Christiansen; it uses find to track down all links and then uses perl to print the names of links that point to nonexistent files. (If you're a Perl hacker and you'll be using this script often, you could replace the Unix find utility with the Perl File::Find module.)

#!/bin/sh

find . -type l -print | perl -nle '-e || print'

The script only lists "dead" links; it doesn't try to delete them or do anything drastic. If you want to take some other action (such as deleting these links automatically), you can use the output of the script in backquotes (Section 28.14). For example:

% rm `oldlinks`

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Shell scripts, aliases, and other programs often need temporary files to hold data to be used later. If the program will be run more than once, or if the temp file needs to stay around after the program is done, you need some way to make a unique filename. Generally these files are stored in /tmp or /usr/tmp.

One way is with the shell's process ID number (Section 24.3), available in the $$ parameter. You might name a file /tmp/ myprog$$; the shell will turn that into something like /tmp/ myprog1234 or /tmp/ myprog28471. If your program needs more than one temporary file, add an informative suffix to the names:

% errs=/tmp/

               myprog-errs$$

% output=/tmp/

               myprog-output$$

You can also use date's + option to get a representation of the date suitable for temporary filenames. For example, to output the Year, month, day, Hour, Minute, and Second:

% date

Wed Mar  6 17:04:39 MST 2002

% date +'%Y%m%d%H%M%S'

20020306170515

Use a + parameter and backquotes ( ``) (Section 28.14) to get a temp file named for the current date and/or time. For instance, on May 31 the following command would store foo.0531 in the Bourne shell variable temp. On December 7, it would store foo.1207:

% temp=foo.`date +'%m%d'`

If you'll be generating a lot of temporary files in close proximity, you can use both the process ID and the date/time:

% output=/tmp/

               myprog$$.`date +'%Y%m%d%H%M%S'`

% echo $output

/tmp/myprog25297.20020306170222

—JP and DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The utility find is one of the most useful and important of the Unix utilities. It finds files that match a given set of parameters, ranging from the file's name to its modification date. In this chapter, we'll be looking at many of the things it can do. As an introduction, here's a quick summary of its features and basic operators:

% find 

               path operators

where path is one or more directories in which find will begin to search and operators (or, in more customary jargon, options) tell find which files you're interested in. The operators are as follows:

-name filename: Find files with the given filename. This is the most commonly used operator. filename may include wildcards, but if it does, they must be quoted to prevent the shell from interpreting the wildcards.
-perm mode: Find files with the given access mode. You must give the access mode in octal.
-type c: Find the files of the given type, specified by c. c is a one-letter code; for example, f for a plain file, b for a block special file, l for a symbolic link, and so forth.
-user name: Find files belonging to user name. name may also be a user ID number.
-group name: Find files belonging to group name. name may also be a group ID number.
-size n: Find files that are n blocks long. A block usually equals 512 bytes. The notation + n says "find files that are over n blocks long." The notation n c says "find files that are n characters long." Can you guess what + n c means?
-inum n: Find files with the inode number n.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The utility find is one of the most useful and important of the Unix utilities. It finds files that match a given set of parameters, ranging from the file's name to its modification date. In this chapter, we'll be looking at many of the things it can do. As an introduction, here's a quick summary of its features and basic operators:

% find 

               path operators

where path is one or more directories in which find will begin to search and operators (or, in more customary jargon, options) tell find which files you're interested in. The operators are as follows:

-name filename: Find files with the given filename. This is the most commonly used operator. filename may include wildcards, but if it does, they must be quoted to prevent the shell from interpreting the wildcards.
-perm mode: Find files with the given access mode. You must give the access mode in octal.
-type c: Find the files of the given type, specified by c. c is a one-letter code; for example, f for a plain file, b for a block special file, l for a symbolic link, and so forth.
-user name: Find files belonging to user name. name may also be a user ID number.
-group name: Find files belonging to group name. name may also be a group ID number.
-size n: Find files that are n blocks long. A block usually equals 512 bytes. The notation + n says "find files that are over n blocks long." The notation n c says "find files that are n characters long." Can you guess what + n c means?
-inum n: Find files with the inode number n.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The first, most obvious, use of this utility is find's ability to locate old, big, or unused files whose locations you've forgotten. In particular, find's most fundamentally important characteristic is its ability to travel down subdirectories.

Normally the shell provides the argument list to a command. That is, Unix programs are frequently given filenames and not directory names. Only a few programs can be given a directory name and march down the directory searching for subdirectories. The programs find, tar (Section 38.3), du, and diff do this. Some versions of chmod (Section 50.5), chgrp, ls, rm, and cp will, but only if a -r or -R option is specified.

In general, most commands do not understand directory structures and rely on the shell to expand wildcards to directory names. That is, to delete all files whose names end with a .o in a group of directories, you could type:

% rm *.o */*.o */*/*.o

Not only is this tedious to type, it may not find all of the files you are searching for. The shell has certain blind spots. It will not match files in directories whose names start with a dot. And, if any files match */*/*/*.o, they would not be deleted.

Another problem is typing the previous command and getting the error "Arguments too long." This means the shell would expand too many arguments from the wildcards you typed.

find is the answer to these problems.

A simple example of find is using it to print the names of all the files in the directory and all subdirectories. This is done with the simple command:

% find . -print

The first arguments to find are directory and file pathnames — in the example, a dot (.) is one name for the current directory. The arguments after the pathnames always start with a minus sign (-) and tell find what to do once it finds a file; these are the search operators. In this case, the filename is printed.

You can use the tilde (~), as well as particular paths. For example:

% find ~ ~barnett /usr/local -print

And if you have a very slow day, you can type:

% find / -print

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

"Why didn't find find my file?" I wondered sometimes. "I know it's there!"

More often than not, I'd forgotten to use -print. Without -print (or -ls, on versions of find that have it), find may not print any pathnames. For a long time, this quirk of find confused new users, so most modern versions of find will assume -print if you don't supply an action; some will give you an error message instead. If you don't get the output you expected from find, check to make sure that you specified the action you meant.

—JP and DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can look for particular files by using an expression with wildcards (Section 28.3) as an argument to the -name operator. Because the shell also interprets wildcards, it is necessary to quote them so they are passed to find unchanged. Any kind of quoting can be used:

% find . -name \*.o -print

% find . -name '*.o' -print

% find . -name "[a-zA-Z]*.o" -print

Any directory along the path to the file is not matched with the -name operator, merely the name at the end of the path. For example, the previous commands would not match the pathname ./subdir.o/afile — but they would match ./subdir.o and ./src/subdir/prog.o.

Section 9.27 shows a way to match directories in the middle of a path. Here's a simpler "find file" alias that can come in very handy:


               alias ff "find . -name '*\!{*}*' -ls"

Give it a file or directory name; the alias will give a long listing of any file or directory names that contain the argument. For example:

% ff ch09

2796156 4 -rw-r--r--  1 deb  deb  628 Feb  2 10:41 ./oreilly/UPT/book/ch09.sgm

—BB and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you want to find a file that is seven days old, use the -mtime operator:

% find . -mtime 7 -print

An alternate way is to specify a range of times:

% find . -mtime +6 -mtime -8 -print

mtime is the last modified time of a file. If you want to look for files that have not been used, check the access time with the -atime argument. Here is a command to list all files that have not been read in 30 days or more:

% find . -type f -atime +30 -print

It is difficult to find directories that have not been accessed because the find command modifies the directory's access time.

There is another time associated with each file, called the ctime, the inode change time. Access it with the -ctime operator. The ctime will have a more recent value if the owner, group, permission, or number of links has changed, while the file itself has not. If you want to search for files with a specific number of links, use the -links operator.

Section 8.2 has more information about these three times, and Section 9.7 explains how find checks them.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

find is admittedly tricky. Once you get a handle on its abilities, you'll learn to appreciate its power. But before thinking about anything remotely tricky, let's look at a simple find command:

% find . -name "*.c" -print

The . tells find to start its search in the current directory (.) and to search all subdirectories of the current directory. The

-name

"*.c"

tells find to find files whose names end in .c. The -print operator tells find how to handle what it finds, i.e., print the names on standard output.

All find commands, no matter how complicated, are really just variations on this one. You can specify many different names, look for old files, and so on; no matter how complex, you're really only specifying a starting point, some search parameters, and what to do with the files (or directories or links or . . . ) you find.

The key to using find in a more sophisticated way is realizing that search parameters are really "logical expressions" that find evaluates. That is, find:

Looks at every file, one at a time.
Uses the information in the file's inode to evaluate an expression given by the command-line operators.
Takes the specified action (e.g., printing the file's name) if the expression's value is "true."

So, -name "*.c" is really a logical expression that evaluates to true if the file's name ends in .c.

Once you've gotten used to thinking this way, it's easy to use the AND, OR, NOT, and grouping operators. So let's think about a more complicated find command. Let's look for files that end in .o or .tmp AND that are more than five days old, AND let's print their pathnames. We want an expression that evaluates true for files whose names match either *.o OR *.tmp:

-name "*.o" -o -name "*.tmp"

If either condition is true, we want to check the access time. So we put the previous expression within parentheses (quoted with backslashes so the shell doesn't treat the parentheses as subshell operators). We also add a -atime operator:

-atime +5 \( -name "*.o" -o -name "*.tmp" \)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The times that go with the find operators -mtime , -atime, and -ctime often aren't documented very well. The times are in days:

A number with no sign, for example, 3 (as in -mtime 3 or -atime 3), means the 24-hour period that ended exactly 3 days ago (in other words, between 96 and 72 hours ago).
A number with a minus sign (-) refers to the period since that 24-hour period. For example, -3 (as in -mtime -3) is any time between now and 3 days ago (in other words, between 0 and 72 hours ago).
Naturally, a number with a plus sign (+) refers to the period before that 24-hour period. For example, +3 (as in -mtime +3) is any time more than 3 days ago (in other words, more than 96 hours ago).

Got that? Then you should see that -atime -2 and -atime 1 are both true on files that have been accessed between 48 and 24 hours ago. (-atime -2 is also true on files accessed 24 hours ago or less.)

For more exact comparisons, use find -newer with touch Section 9.8).

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One problem with find's time operators (-atime and its brethren) is that they don't allow very exact comparisons. They only allow you to specify time to within a day, and sometimes that's just not good enough. You think that your system was corrupted at roughly 4 p.m. yesterday (March 20); you want to find any files that were modified after that point, so you can inspect them. Obviously, you'd like something more precise than "give me all the files that were modified in the last 24 hours."

Some versions of touch , and other freely available commands like it, can create a file with an arbitrary timestamp. That is, you can use touch to make a file that's backdated to any point in the past (or, for that matter, postdated to some point in the future). This feature, combined with find's -newer operator, lets you make comparisons accurate to one minute or less.

For example, to create a file dated 4 p.m., March 20, give the command:

% touch -t 03201600 /tmp/4PMyesterday

Then to find the files created after this, give the command:

% find . -newer /tmp/4PMyesterday -print

What about "older" files? Older files are "not newer" files, and find has a convenient NOT operator (!) for just this purpose. So let's say that you want to find files that were created between 10:46 a.m. on July 3, 1999 and 9:37 p.m. on June 4, 2001. You could use the following commands:

% touch -t 199907031046 /tmp/file1

% touch -t 200106042137 /tmp/file2

% find . -newer /tmp/file1 \! -newer /tmp/file2 -print

% rm /tmp/file[12]

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Often, when you find a file, you don't just want to see its name; you want to do something, like grep (Section 13.2) for a text string. To do this, use the -exec operator. This allows you to specify a command that is executed upon each file that is found.

The syntax is peculiar and in many cases, it is simpler just to pipe the output of find to xargs (Section 28.17). However, there are cases where -exec is just the thing, so let's plunge in and explain its peculiarities.

The -exec operator allows you to execute any command, including another find command. If you consider that for a moment, you realize that find needs some way to distinguish the command it's executing from its own arguments. The obvious choice is to use the same end-of-command character as the shell (the semicolon). But since the shell uses the semicolon itself, it is necessary to escape the character with a backslash or quotes.

Therefore, every -exec operator ends with the characters \;. There is one more special argument that find treats differently: {} . These two characters are used as the variable whose name is the file find found. Don't bother rereading that last line: an example will clarify the usage. The following is a trivial case and uses the -exec operator with echo to mimic the -print operator:

% find . -exec echo {} \;

The C shell ( Section 29.1) uses the characters { and }, but doesn't change {} together, which is why it is not necessary to quote these characters. The semicolon must be quoted, however. Quotes can be used instead of a backslash:

% find . -exec echo {} ';'

as both will sneak the semicolon past the shell and get it to the find command. As I said before, find can even call find. If you wanted to list every symbolic link in every directory owned by a group staff under the current directory, you could execute:

% find `pwd` -type d -group staff -exec find {} -type l -print \;

To search for all files with group-write permission under the current directory and to remove the permission, you can use:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's something that will really make your head spin. Remember that -exec doesn't necessarily evaluate to "true"; it only evaluates to true if the command it executes returns a zero exit status. You can use this to construct custom find tests.

Assume that you want to list files that are "beautiful." You have written a program called beauty that returns zero if a file is beautiful and nonzero otherwise. (This program can be a shell script, a perl script, an executable from a C program, or anything you like.)

Here's an example:

% find . -exec beauty {} \; -print

In this command, -exec is just another find operator. The only difference is that we care about its value; we're not assuming that it will always be "true." find executes the beauty command for every file. Then -exec evaluates to true when find is looking at a "beautiful" program, causing find to print the filename. (Excuse us, causing find to evaluate the -print. :-))

Of course, this ability is capable of infinite variation. If you're interested in finding beautiful C code, you could use the command:

% find . -name "*.[ch]" -exec beauty {} \; -print

For performance reasons, it's a good idea to put the -exec operator as close to the end as possible. This avoids starting processes unnecessarily; the -exec command will execute only when the previous operators evaluate to true.

—JP and ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

My favorite reason to use find 's -exec is for large recursive greps. Let's say I want to search through a large directory with lots of subdirectories to find all of the .cc files that call the method GetRaw( ):

% find . -name \*.cc -exec grep -n "GetRaw(" {} \; -print

58:    string Database::GetRaw(const Name &owner) const {

67:    string Database::GetRaw(const Name &owner,

./db/Database.cc

39:            return new Object(owner, _database->GetRaw(owner));

51:    string Object::GetRaw(const Property& property) const {

52:        return _database->GetRaw(_owner, property);

86:            Properties properties(_database->GetRaw(owner));

103:        return _database->GetRaw(_owner);

./db/Object.cc

71:        return new DatabaseObject(owner, GetDatabase( ).GetRaw(owner));

89:            return Sexp::Parse(GetRaw(property));

92:            SexpPtr parent = Sexp::Parse(GetRaw("_parent"))->Eval(this);

./tlisp/Object.cc

This output is from a real source directory for an open source project I'm working on; it shows me each line that matched my grep along with its line number, followed by the name of the file where those lines were found. Most versions of grep can search recursively (using -R), but they search all files; you need find to grep through only certain files in a large directory tree.

—JP and DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Running find is fairly time consuming, and for good reason: it has to read every inode in the directory tree that it's searching. Therefore, combine as many things as you can into a single find command. If you're going to walk the entire tree, you may as well accomplish as much as possible in the process.

Let's work from an example. Assume that you want to write a command (eventually for inclusion in a Chapter 27 shell script) that sets file-access modes correctly. You want to give 771 access to all directories, 600 access for all backup files (*.BAK), 755 access for all shell scripts (*.sh), and 644 access for all text files (*.txt). You can do all this with one command:

$ find . \( -type d       -a -exec chmod 771 {} \; \) -o \

                        \( -name "*.BAK" -a -exec chmod 600 {} \; \) -o \

                        \( -name "*.sh"  -a -exec chmod 755 {} \; \) -o \

                        \( -name "*.txt" -a -exec chmod 644 {} \; \)

Why does this work? Remember that -exec is really just another part of the expression; it evaluates to true when the following command is successful. It isn't an independent action that somehow applies to the whole find operation. Therefore, -exec can be mixed freely with -type, -name, and so on.

However, there's another important trick here. Look at the first chunk of the command — the first statement, that is, between the first pair of $ and $. It says, "If this file is a directory and the chmod command executes successfully . . . " Wait. Why doesn't the -exec execute a chmod on every file in the directory to see whether it's successful?

Logical expressions are evaluated from left to right; in any chunk of the expression, evaluation stops once it's clear what the outcome is. Consider the logical expression "`A AND B' is true." If A is false, you know that the result of "`A AND B' is true" will also be false — so there's no need to look the rest of the statement, B.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you are only interested in files of a certain type, use the -type argument, followed by one of the characters in Table 9-1. Note, though that some versions of find don't have all of these.

Table 9-1: find -type characters
Character	Meaning
`b`	Block special file ("device file")
`c`	Character special file ("device file")
`d`	Directory
`f`	Plain file
`l`	Symbolic link
`p`	Named pipe file
`s`	Socket

Unless you are a system administrator, the important types are directories, plain files, or symbolic links (i.e., types d, f, or l).

Using the -type operator, here is another way to list files recursively:

% find . -type f -print | xargs ls -l

It can be difficult to keep track of all the symbolic links in a directory. The next command will find all the symbolic links in your home directory and print the files to which your symbolic links point.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

find has several operators that take a decimal integer. One such argument is -size. The number after this argument is the size of the files in disk blocks. Unfortunately, this is a vague number. Earlier versions of Unix used disk blocks of 512 bytes. Newer versions allow larger block sizes, so a "block" of 512 bytes is misleading.

This confusion is aggravated when the command ls -s is used. The -s option supposedly lists the size of the file in blocks. But if your system has a different block size than ls -s has been programmed to assume, it can give a misleading answer. You can put a c after the number and specify the size in bytes. To find a file with exactly 1,234 bytes (as in an ls -l listing), type:

% find . -size 1234c -print

To search for files using a range of file sizes, a minus or plus sign can be specified before the number. The minus sign (-) means less than, and the plus sign (+) means greater than. This next example lists all files that are greater than 10,000 bytes, but less than 32,000 bytes:

% find . -size +10000c -size -32000c -print

When more than one qualifier is given, both must be true.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

find can look for files with specific permissions. It uses an octal number for these permissions. If you aren't comfortable with octal numbers and the way Unix uses them in file permissions, Section 1.17 is good background reading.

The string rw-rw-r-- indicates that you and members of your group have read and write permission, while the world has read-only privilege. The same permissions are expressed as an octal number as 664. To find all *.o files with these permissions, use the following:

% find . -name \*.o -perm 664 -print

To see if you have any directories with write permission for everyone, use this:

% find . -type d -perm 777 -print

The previous examples only match an exact combination of permissions. If you wanted to find all directories with group write permission, you want to match the pattern ----w----. There are several combinations that can match. You could list each combination, but find allows you to specify a pattern that can be bitwise ANDed with the permissions of the file. Simply put a minus sign (-) before the octal value. The group write permission bit is octal 20, so the following negative value:

% find . -perm -20 -print

will match the following common permissions:

Permission	Octal value
`rwxrwxrwx`	777
`rwxrwxr-x`	775
`rw-rw-rw-`	666
`rw-rw-r--`

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Often you need to look for a file belonging to a certain user or group. This is done with the -user and -group search operators. You often need to combine this with a search for particular permissions. To find all files that are set user ID (setuid) root, use this:

% find . -user root -perm -4000 -print

To find all files that are set group ID (setgid) staff, use this:

% find . -group staff -perm -2000 -print

Instead of using a name or group from /etc/passwd or /etc/group, you can use the UID or GID number:

% find . -user 0 -perm -4000 -print

% find . -group 10 -perm -2000 -print

Often, when a user leaves a site, his account is deleted, but his files are still on the computer. Some versions of find have -nouser or -nogroup operators to find files with an unknown user or group ID.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In many versions of find, the operator {} , used with the -exec operator, only works when it's separated from other arguments by whitespace. So, for example, the following command will not do what you thought it would:

% find . -type d -exec mkdir /usr/project/{} \;

You might have thought this command would make a duplicate set of (empty) directories, from the current directory and down, starting at the directory /usr/project. For instance, when the find command finds the directory ./adir, you would have it execute mkdir /usr/project/./adir (mkdir will ignore the dot; the result is /usr/project/adir).

That doesn't work because those versions of find don't recognize the {} in the pathname. The GNU version does expand {} in the middle of a string. On versions that don't, though, the trick is to pass the directory names to sed , which substitutes in the leading pathname:

% find . -type d -print | sed 's@^@/usr/project/@' | xargs mkdir

% find . -type d -print | sed 's@^@mkdir @' | (cd /usr/project; sh)

Let's start with the first example. Given a list of directory names, sed substitutes the desired path to that directory at the beginning of the line before passing the completed filenames to xargs and mkdir. An @ is used as a sed delimiter because slashes (/) are needed in the actual text of the substitution. If you don't have xargs, try the second example. It uses sed to insert the mkdir command, then it changes to the target directory in a subshell where the mkdir commands will actually be executed.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Berkeley added a handy feature to its find command — if you give it a single argument, it will search a database for file or directory names that match. For example, if you know there's a file named MH.eps somewhere on the computer but you don't know where, type the following:

% find MH.eps

/nutshell/graphics/cover/MH.eps

That syntax can be confusing to new users: you have to give find just one argument. With more arguments, find searches the filesystem directly. Maybe that's one reason that GNU has a "fast find" utility named locate — and its find utility always searches, as described in the rest of this chapter. The GNU slocate command is a security-enhanced version of locate. In the rest of this article, I'll describe locate — but find with a single argument (as shown previously) works about the same way.

The "fast find" database is usually rebuilt every night. So, it's not completely up-to-date, but it's usually close enough. If your system administrator has set this up, the database usually lists all files on the filesystem — although it may not list files in directories that don't have world-access permission. If the database isn't set up at all, you'll get an error like /usr/lib/find/find.codes: No such file or directory. (If that's the case, you can set up a "fast find" database yourself. Set up your own private locate database, or see Section 9.20.)

Unless you use wildcards, locate does a simple string search, like fgrep, through a list of absolute pathnames. Here's an extreme example:

% locate bin

/bin

/bin/ar

   ...

/home/robin

/home/robin/afile

/home/sally/bin

   ...

You can cut down this output by piping it through grep, sed, and so on. But locate and "fast find" also can use wildcards to limit searches. Section 9.19 explains this in more detail.

locate has an advantage over the "fast find" command: you can have multiple file databases and you can search some or all of them. locate and slocate

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

locate and all the "fast find" commands I've used can match shell wildcards ( Section 1.13) (* , ?, [ ]). If you use a wildcard on one end of the pattern, the search pattern is automatically "anchored" to the opposite end of the string (the end where the wildcard isn't). The shell matches filenames in the same way.

The difference between the shell's wildcard matching and locate matching is that the shell treats slashes (/ ) in a special manner: you have to type them as part of the expression. In locate, a wildcard matches slashes and any other character. When you use a wildcard, be sure to put quotes around the pattern so the shell won't touch it.

Here are some examples:

To find any pathname that ends with bin:

% locate '*bin'

/bin

/home/robin

/home/robin/bin

   ...

To find any pathname that ends with /bin (a good way to find a file or directory named exactly bin):
```
% locate '*/bin'

/bin

/home/robin/bin

/usr/bin

   ...
```
Typing locate '*bin*' is the same as typing locate bin.
To match the files in a directory named bin, but not the directory itself, try something like this:
```
% locate '*/bin/*'

/bin/ar

/bin/cat

   ...

/home/robin/bin/prog
```
To find the files in /home whose names end with a tilde (~) (these are probably backup files from the Emacs editor):
```
% locate '/home/*~'

/home/testfile~

/home/allan/.cshrc~

/home/allan/.login~

/home/dave/.profile~

   ...
```
Notice that the locate asterisk matches dot files, unlike shell wildcards.
The question mark (?) and square brackets ([ ]) operators work, too. They're not quite as useful as they are in the shell because they match the slashes (/) in the pathnames. Here are a couple of quick examples:
```
% locate '????'

/bin

/etc

/lib

/src

/sys

/usr

% locate '/[bel]??'

/bin

/etc

/lib
```

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you use find to search for files, you know that it can take a long time to work, especially when there are lots of directories to search. Here are some ideas for speeding up your finds.

By design, setups like these that build a file database won't have absolutely up-to-date information about all your files.

If your system has " fast find" or locate, that's probably all you need. It lets you search a list of all pathnames on the system.

Even if you have "fast find" or locate, it still might not do what you need. For example, those utilities only search for pathnames. To find files by the owner's name, the number of links, the size, and so on, you have to use "slow find." In that case — or, when you don't have "fast find" or locate — you may want to set up your own version.

slocate can build and update its own database (with its -u option), as well as search the database. The basic "fast find" has two parts. One part is a command, a shell script usually named updatedb or locate.updatedb, that builds a database of the files on your system — if your system has it, take a look to see a fancy way to build the database. The other part is the find or locate command itself — it searches the database for pathnames that match the name (regular expression) you type.

To make your own "fast find":

Pick a filename for the database. We'll use $HOME/.fastfind (some systems use $LOGDIR instead of $HOME).
Design the find command you want to use. The command to build a database of all the files in your home directory might look like this:
```
% cd

% find . -print | sed "s@^./@@" > .fastfind.new

% mv -f .fastfind.new .fastfind

                  
```
That doesn't update the database until the new one is finished. It also doesn't compress the database. If you're short on disk space, use this instead:
```
% cd

% find . -print | sed "s@^./@@" | gzip > .fastfind.gz

                  
```
The script starts from your home directory, then uses sed (Section 13.9) to strip the start of the pathname (like

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Want to search every file, in some directory and all its subdirectories, to find the file that has a particular word or string in it? That's a job for find and one of the grep commands.

For example, to search all the files for lines starting with a number and containing the words "SALE PRICE," you could use:

% egrep '^[0-9].*SALE PRICE' `find . -type f -print`

./archive/ad.1290: 1.99 a special SALE PRICE

./archive/ad.0191: 2.49 a special SALE PRICE

Using the backquotes (``) might not work. If find finds too many files, egrep 's command-line arguments can get too long. Using xargs can solve that; it splits long sets of arguments into smaller chunks. There's a problem with that: if the last "chunk" has just one filename and the grep command finds a match there, grep won't print the filename:

% find . -type f -print | xargs fgrep '$12.99'

./old_sales/ad.0489: Get it for only $12.99!

./old_sales/ad.0589: Last chance at $12.99, this month!

Get it for only $12.99 today.

The answer is to add the Unix " empty file," /dev/null. It's a filename that's guaranteed never to match but always to leave fgrep with at least two filenames:

% find . -type f -print | xargs fgrep '$12.99' /dev/null

Then xargs will run commands like these:

fgrep '$12.99' /dev/null ./afile ./bfile ...

fgrep '$12.99' /dev/null ./archives/ad.0190 ./archives/ad.0290 ...

fgrep '$12.99' /dev/null ./old_sales/ad.1289

That trick is also good when you use a wildcard (Section 28.3) and only one file might match it. grep won't always print the file's name unless you add /dev/null:

% grep "whatever" /dev/null /x/y/z/a*

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The following simple shell script, lookfor, uses find to look for all files in the specified directory hierarchy that have been modified within a certain time, and it passes the resulting names to grep to scan for a particular pattern. For example, the command:

% lookfor /work -7 tamale enchilada

would search through the entire /work filesystem and print the names of all files modified within the past week that contain the words "tamale" or "enchilada." (For example, if this article is stored in /work, lookfor should find it.)

The arguments to the script are the pathname of a directory hierarchy to search in ($1), a time ($2), and one or more text patterns (the other arguments). This simple but slow version will search for an (almost) unlimited number of words:

#!/bin/sh

temp=/tmp/lookfor$$

trap 'rm -f $temp; exit' 0 1 2 15

find $1 -mtime $2 -print > $temp

shift; shift

for word

do grep -i "$word" `cat $temp` /dev/null

done

That version runs grep once to search for each word. The -i option makes the search find either upper- or lowercase letters. Using /dev/null makes sure that grep will print the filename. Watch out, though: the list of filenames may get too long.

The next version is more limited but faster. It builds a regular expression for egrep that finds all the words in one pass through the files. If you use too many words, egrep will say Regular expression too long. Also, your egrep may not have a -i option; you can just omit it. This version also uses xargs; though xargs has its problems.

#!/bin/sh

where="$1"

when="$2"

shift; shift

# Build egrep expression like (word1|word2|...) in $expr

for word

do

    case "$expr" in

    "") expr="($word" ;;

    *) expr="$expr|$word" ;;

    esac

done

expr="$expr)"

  

find $where -mtime $when -print | xargs egrep -i "$expr" /dev/null

—JP and TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Even a graphical file manager might not be enough to help you step through a complicated directory tree with multiple layers of subdirectories. Which directories have you visited so far, and which are left to go? This article shows a simple way, using shell arrays, to step through a tree directory-by-directory. The technique is also good for stepping through lists of files — or almost any collection of things, over a period of time — of which you don't want to miss any. At the end are a couple of related tips on using arrays.

Let's start with a quick overview of expanding array values; then we'll look at specifics for each shell. A dollar sign ($) before the name of a shell variable gives you its value. In the C shells and zsh, that gives all members of an array. But, in the Korn shell and bash2, expanding an array value without the index gives just the first member. To pick out a particular member, put its number in square brackets after the name; in ksh and bash2, you also need to use curly braces ({}). A hash mark (#) gives the number of members. Finally, you can use range operators to choose several members of an array.

Here's a practical example that you might use, interactively, at a shell prompt. You're cleaning your home directory tree. You store all the directory names in an array named d. When you've cleaned one directory, you go to the next one. This way, you don't miss any directories. (To keep this simple, I'll show an example with just four directories.)

If you don't want to use shell commands to browse the directories, you could use a command to launch a graphical file browser on each directory in the array. For instance, make the nextdir alias launch Midnight Commander with mc $d[1].

Let's start with the C shell:

% set d=(`find $home -type d -print`)

% echo $#d directories to search: $d

4 directories to search: /u/ann /u/ann/bin /u/ann/src /u/ann/lib

% alias nextdir 'shift d; cd $d[1]; pwd; ls -l'

% cd $d[1]

   ...clean up first directory...

% nextdir

/u/ann/bin

total 1940

lrwxrwxrwx    1 ann    users      14 Feb  7  2002 ] -> /usr/ucb/reset

-r-xr-xr-x    1 ann    users    1134 Aug 23  2001 addup

   ...clean up bin directory...

%

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here is how to find hard links, as well as a brief look at the Unix filesystem from the user's viewpoint. Suppose you are given the following:

% ls -li /usr/bin/at

8041 -r-sr-xr-x  4 root  wheel  19540 Apr 21  2001 /usr/bin/at*

In other words, there are four links, and /usr/bin/at is one of four names for inode 8041. You can find the full names of the other three links by using find. However, just knowing the inode number does not tell you everything. In particular, inode numbers are only unique to a given filesystem. If you do a find / -inum 8041 -print , you may find more than four files, if inode 8041 is also on another filesystem. So how do you tell which ones refer to the same file as /usr/bin/at?

The simplest way is to figure out the filesystem on which /usr/bin/at lives by using df:

% df /usr/bin/at

Filesystem   1K-blocks     Used    Avail Capacity  Mounted on

/dev/ad0s1f    3360437  1644024  1447579    53%    /usr

Then start your find at the top of that filesystem, and use -xdev to tell it not to search into other filesystems:

% find /usr -xdev -inum 8041 -print

/usr/bin/at

/usr/bin/atq

/usr/bin/atrm

/usr/bin/batch

Some manpages list -x as an alternative to -xdev; -xdev is generally more portable.

—DJPH and CT

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

find has lots of operators for finding some particular kinds of files. But find won't stop at your current directory — if there are subdirectories, it looks there too. How can you tell it "only the current directory"? Use -prune.

Most finds also have a -maxdepth option that gives the maximum number of directory levels to descend. For example, find . -maxdepth 0 operates only on the current directory.

-prune cuts short find's search at the current pathname. So, if the current pathname is a directory, find won't descend into that directory for any further searches. The command line looks kind of hairy. Here's one to find all files modified in the last 24 hours from the current directory:

% date

Tue Feb 12 19:09:35 MST 2002

% ls -l

total 0

drwxr-xr-x  1 deb  deb  0 Feb 12 12:11 adir

-rw-r--r--  1 deb  deb  0 Feb 12 19:08 afile

-rw-r--r--  1 deb  deb  0 Jan 10 10:37 bfile

-rw-r--r--  1 deb  deb  0 Feb 11 22:43 cfile

% find . \( -type d ! -name . -prune \) -o \( -mtime -1 -print \)

./afile

./cfile

Let's try to understand this command: once you see the pattern, you'll understand some important things about find that many people don't. Let's follow find as it looks at a few pathnames.

find looks at each entry, one by one, in the current directory (.). For each entry, find tries to match the expression from left to right. As soon as some parenthesized part matches, it ignores the rest (if any) of the expression.

When find is looking at the file named ./afile, the first part of the expression, ( -type d ! -name . -prune ), doesn't match (./afile isn't a directory). So find doesn't prune. It tries the other part, after the -o (or):

Has ./afile been modified in the last day? In this (imaginary) case, it has — so the -print (which is always true) prints the pathname.

Next, ./bfile: like the previous step, the first part of the expression won't match. In the second part, ( -mtime -1 -print ), the file's modification time is more than one day ago. So the -mtime -1 part of the expression is false;

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

find -prune prunes find's search tree at the current pathname. Here are a couple of aliases that use -prune to search for files in the current directory. The first one, named find. (with a dot on the end of its name, to remind you of ., the relative pathname for the current directory), simply prints names with -print. The second alias gives a listing like ls -gilds. You can add other find operators to the command lines to narrow your selection of files. The aliases work like this:

% find. -mtime -1

./afile

./cfile

% find.ls -mtime -1

43073   0 -r--------  1 jerry    ora        0 Mar 27 18:16 ./afile

43139   2 -r--r--r--  1 jerry    ora     1025 Mar 24 02:33 ./cfile

The find. alias is handy inside backquotes, feeding a pipe, and other places you need a list of filenames. The second one, find.ls, uses -ls instead of -print:

alias find. 'find . \( -type d ! -name . -prune \) -o \( \!* -print \)'

alias find.ls 'find . \( -type d ! -name . -prune \) -o \( \!* -ls \)'

If you don't want the ./ at the start of each name, add a pipe through cut -c3- or cut -d'/' -f2- to the end of the alias definition.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Q: I want to run find across a directory tree, skipping standard directories like /usr/spool and /usr/local/bin. A -name dirname -prune clause won't do it because -name doesn't match the whole pathname — just each part of it, such as spool or local. How can I make find match the whole pathname, like /usr/local/bin/, instead of all directories named bin?

A: It cannot be done directly. You can do this:

% find /path -exec test {} = /foo/bar -o {} = /foo/baz \; -prune -o pred

This will not perform pred on /foo/barand /foo/baz; if you want them done, but not any files within them, try:

% find /path \( -exec test test-exprs \; ! -prune \) -o pred

The second version is worth close study, keeping the manual for find at hand for reference. It shows a great deal about how find works.

The -prune operator simply says "do not search the current path any deeper" and then succeeds a la -print.

Q: I only want a list of pathnames; the pred I use in your earlier answer will be just -print. I think I could solve my particular problem by piping the find output through a sed or egrep -v filter that deletes the pathnames I don't want to see.

A: That would probably be fastest. Using test runs the test program for each file name, which is quite slow. Take a peek at locate, described in Section 9.18.

There's more about complex find expressions in other articles, especially Section 9.6 and Section 9.12.

—CT and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The most painful aspect of a large NFS environment is avoiding the access of files on NFS servers that are down. find is particularly sensitive to this because it is very easy to access dozens of machines with a single command. If find tries to explore a file server that happens to be down, it will time out. It is important to understand how to prevent find from going too far.

To do this, use -xdev or -prune with -fstype, though, unfortunately, not all finds have all of these. -fstype tests for filesystem types and expects an argument like nfs, ufs, cd9660, or ext2fs. To limit find to files only on a local disk or disks, use the clause -fstype nfs -prune, or, if your find supports it, -fstype local.

To limit the search to one particular disk partition, use -xdev. For example, if you need to clear out a congested disk partition, you could look for all files bigger than 10 MB (10*1024*1024) on the disk partition containing /usr, using this command:

% find /usr -size +10485760c -xdev -print

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

At first glance, there doesn't seem to be enough material to fill an entire chapter with information about linking, moving, and copying files. However, there are several things that make the topic more complex (and more interesting) than you might expect:

In addition to moving and copying files, Unix systems also allow you to link them — to have two filenames, perhaps in different directories or even on different filesystems, that point to the same file. Section 10.3 explores the reasons why you want to do that; Section 10.4 discusses the difference between "hard" and "soft" links; Section 10.5 demonstrates how to create links; and other articles discuss various issues that can come up when using links.
It's nontrivial to rename a group of files all at once, but Unix provides many ways to circumvent the tedium of renaming files individually. In the chapter you'll see many different ways to do this, exploring the variety in the Unix toolbox along the way.
In a hierarchical filesystem, you're sometimes faced with the problem of moving not only files but entire directory hierarchies from one place to another. Section 10.12 and Section 10.13 demonstrate two techniques you can use to perform this task.

— TOR

Before you can understand moving and copying files, you need to know a bit more about how files are represented in directories. What does it mean to say that a file is really "in" a directory? It's easy to imagine that files are actually inside of something (some special chunk of the disk that's called a directory). But that's precisely wrong, and it's one place where the filing cabinet model of a filesystem doesn't apply.

A directory really is just another file, and it really isn't different from any other datafile. If you want to prove this, try the command od -c . On some Unix systems, it dumps the current directory to the screen in raw form. The result certainly looks ugly (it's not a text file; it just has lots of binary characters). But, if your system allows it,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

At first glance, there doesn't seem to be enough material to fill an entire chapter with information about linking, moving, and copying files. However, there are several things that make the topic more complex (and more interesting) than you might expect:

In addition to moving and copying files, Unix systems also allow you to link them — to have two filenames, perhaps in different directories or even on different filesystems, that point to the same file. Section 10.3 explores the reasons why you want to do that; Section 10.4 discusses the difference between "hard" and "soft" links; Section 10.5 demonstrates how to create links; and other articles discuss various issues that can come up when using links.
It's nontrivial to rename a group of files all at once, but Unix provides many ways to circumvent the tedium of renaming files individually. In the chapter you'll see many different ways to do this, exploring the variety in the Unix toolbox along the way.
In a hierarchical filesystem, you're sometimes faced with the problem of moving not only files but entire directory hierarchies from one place to another. Section 10.12 and Section 10.13 demonstrate two techniques you can use to perform this task.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Before you can understand moving and copying files, you need to know a bit more about how files are represented in directories. What does it mean to say that a file is really "in" a directory? It's easy to imagine that files are actually inside of something (some special chunk of the disk that's called a directory). But that's precisely wrong, and it's one place where the filing cabinet model of a filesystem doesn't apply.

A directory really is just another file, and it really isn't different from any other datafile. If you want to prove this, try the command od -c . On some Unix systems, it dumps the current directory to the screen in raw form. The result certainly looks ugly (it's not a text file; it just has lots of binary characters). But, if your system allows it, od - c should let you see the names of the files that are in the current directory [and, probably, some names of files that have been deleted! Sorry, they're only the old directory entries; you can't get the files back — JP]. If od - c . doesn't work (and it won't on current versions of Linux, for example), use ls - if instead.

A directory is really just a list of files represented by filenames and inode numbers, as shown in the output in Example 10-1.

Example 10-1. Directory-content visualization

The file named    .          is inode 34346

The file named    ..         is inode 987

The file named    mr.ed      is inode 10674

The file named    joe.txt    is inode 8767

The file named    grok       is inode 67871

The file named    otherdir   is inode 2345

When you give a filename like grok, the kernel looks up grok in the current directory and finds out that this file has inode 67871; it then looks up this inode to find out who owns the file, where the data blocks are, and so on.

What's more, some of these "files" may be directories in their own right. In particular, that's true of the first two entries: . and ... These entries are in

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We've talked about hard links (Section 10.1) and symbolic links in a number of places, but we've not discussed why you'd want a file with several names. It was easy to understand what a link would do, but why would you want one?

There are many situations that links (and only links) are able to handle. Once you've seen a few of the problems that a link can solve, you'll start seeing even more situations in which they are appropriate.

Consider a company phone list on a system that is shared by several users. Every user might want a copy of the phone list in his home directory. However, you wouldn't want to give each user a different phone list. In addition to wasting disk space, it would be a pain to modify all of the individual lists whenever you made a change. Giving each user a "link" to a master phone list is one way to solve the problem.

Similarly, assume that you use several different systems that share files via NFS. Eventually, you get tired of editing five or six different .login and .cshrc files whenever you decide to add a new alias or change some element in your startup file; you'd like to have the exact same file appear in each of your home directories. You might also want to give several systems access to the same master database files.

How about this: you have a program or script that performs several related functions. Why not perform them all with the same executable? The script or program just needs to check the name by which it's called and act accordingly.

As another example, assume that you have two versions of a file: a current version, which changes from time to time, and one or more older versions. One good convention would be to name the files data. date, where date shows when the file was created. For example, you might have the files data.jul1, data.jul2, data.jul5, and so on. However, when you access these files, you don't necessarily want to figure out the date — not unless you have a better chronological sense than I do. To make it easier on yourself, create a link (either symbolic or hard) named

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix provides two different kinds of links:

Hard links: With a hard link, two filenames (i.e., two directory entries) point to the same inode and the same set of data blocks. All Unix versions support hard links. They have two important limitations: a hard link can't cross a filesystem (i.e., both filenames must be in the same filesystem), and you can't create a hard link to a directory (i.e., a directory can only have one name). They have two important advantages: the link and the original file are absolutely and always identical, and the extra link takes no disk space (except an occasional extra disk block in the directory file).
Symbolic links (also called soft links or symlinks): With a symbolic link, there really are two different files. One file contains the actual data; the other file just contains the name of the first file and serves as a "pointer." We call the pointer the link. The system knows that whenever it opens a symlink, it should read the contents of the link and then access the file that really holds the data you want. Nearly all Unix systems support symbolic links these days. Symbolic links are infinitely more flexible than hard links. They can cross filesystems or even computer systems (if you are using NFS or RFS (Section 44.9)). You can make a symbolic link to a directory. A symbolic link has its own inode and takes a small amount of disk space to store.

You obviously can't do without copies of files: copies are important whenever users need their own "private version" of some master file. However, links are equally useful. With links, there's only one set of data and many different names that can access it. Section 10.5 shows how to make links.

With a hard link, the two filenames are identical in every way. You can delete one without harming the other. The system deletes the directory entry for one filename and leaves the data blocks (which are shared) untouched. The only thing

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The ln command creates both hard and soft (symbolic) links Section 10.4). If by some strange chance you're using Minix or some other Unix that doesn't have symlinks, then ln won't have the -s option.

% ln 

               filename linkname

                            

                 . . . To create a hard link

% ln -s 

               filename linkname

                         

                . . . To create a symbolic link

If creating a hard link, filename must already exist, or you will get an error message. On many versions of ln, linkname must not exist — if it does, you will also get an error. On other versions, linkname may already exist; if you are allowed to write the file, ln destroys its old contents and creates your link. If you don't have write access for linkname, ln asks whether it is okay to override the file's protection. For example:

% ln foo bar

ln: override protection 444 for bar? y

Typing y gives ln permission to destroy the file bar and create the link. Note that this will still fail if you don't have write access to the directory.

You are allowed to omit the linkname argument from the ln command. In this case, ln takes the last component of filename (i.e., everything after the last slash) and uses it for linkname. Of course, this assumes that filename doesn't refer to the current directory. If it does, the command fails because the link already exists. For example, the following commands are the same:

.. Section 1.16

% ln -s ../archive/file.c file.c

% ln -s ../archive/file.c

Both create a link from file.c in the current directory to ../archive/file.c. ln also lets you create a group of links with one command, provided that all of the links are in the same directory. Here's how:

% ln file1 file2 file3 ... filen directory

This command uses the filename from each pathname (after the last slash) as each link's name. It then creates all the links within the given directory. For example, the first of the following commands is equivalent to the next two:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Symbolic links ( Section 10.5) have one problem. Like good bread, they become "stale" fairly easily. What does that mean?

Consider the following commands:

% ln -s foo bar

% rm foo

What happens if you run these two commands? Remember that the link bar is a pointer: it doesn't have any real data of its own. Its data is the name of the file foo. After deleting foo, the link bar still exists, but it points to a nonexistent file. Commands that refer to bar will get a confusing error message:

% cat bar

cat: bar: No such file or directory

This will drive you crazy if you're not careful. The ls command will show you that bar still exists. You won't understand what's going on until you realize that bar is only a pointer to a file that no longer exists.

The commands ls -Ll or ls -LFwill show an unconnected symbolic link. The -L option means "list the file that this link points to instead of the link itself." If the link points nowhere, ls -L will still list the link.

There are many innocuous ways of creating invalid symbolic links. For example, you could simply mv the data file foo. Or you could move foo, bar, or both to some other part of the filesystem where the pointer wouldn't be valid anymore.

One way to avoid problems with invalid links is to use relative pathnames Section 1.16) when appropriate. For instance, using relative pathnames will let you move entire directory trees around without invalidating links (provided that both the file and the link are in the same tree). Here's an example: assume that you have the file /home/mars/john/project/datastash/input123.txt. Assume that you want to link this file to /home/mars/john/test/input.txt. You create a link by giving the command:

% cd /home/mars/john/test

% ln -s ../project/datastash/input123.txt input.txt

At some later date, you hand the project over to mary, who copies Section 10.13) the entire project and test data trees into her home directory. The link between input.txt

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One feature of symbolic links ( Section 10.5) (a.k.a. symlinks) is that unlike hard links, you can use symbolic links to link directories as well as files. Since symbolic links can span between filesystems, this can become enormously useful.

For example, sometimes administrators want to install a package in a directory tree that's not where users and other programs expect it to be. On our site, we like to keep /usr/bin pure — that is, we like to be sure that all the programs in /usr/bin came with the operating system. That way, when we install a new OS, we know for sure that we can overwrite the entirety of /usr/bin and not lose any "local" programs. We install all local programs in /usr/local.

The X11 package poses a problem, though. Our X windows package (discussed in Chapter 5) expects X11 programs to be installed in /usr/bin/X11. But X isn't distributed as part of our OS, so we'd prefer not to put it there. Instead, we install X programs in /usr/local/X11/bin and create a symbolic link named /usr/bin/X11. We do the same for /usr/include/X11 and /usr/lib/X11:

# ln -s /usr/local/X11/bin /usr/bin/X11

# ln -s /usr/local/X11/lib /usr/lib/X11

# ln -s /usr/local/X11/include /usr/include/X11

By using symlinks, we installed the package where we wanted, but we kept it invisible to any users or programs that expected the X programs, libraries, or include files to be in the standard directories.

Directory links can result in some unexpected behavior, however. For example, let's suppose I want to look at files in /usr/bin/X11. I can just cd to /usr/bin/X11, even though the files are really in /usr/local/X11/bin:

% cd /usr/bin/X11

% ls -F

       mkfontdir*      xcalc*          xinit*          xset*

   ...

But when I do a pwd, I see that I'm really in /usr/local/X11/bin. If I didn't know about the symlink, this might be confusing for me:

% pwd

/usr/local/X11/bin

Now suppose I want to look at files in /usr/bin. Since I did a cd to /usr/bin/X11, I might think I can just go up a level. But that doesn't work:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: sl

The sl program is a perl script (see coverage of Perl in Chapter 41) that traverses the pathnames supplied on the command line, and for each one, it tells you if it had to follow any symbolic links to find the actual filename. Symbolic links to absolute pathnames start over at the left margin. Symbolic links to relative pathnames are aligned vertically with the path element they replace. For example:

$ sl /usr/lib/libXw.a



/usr/lib/libXw.a:

/usr/lib/libXw.a -> /usr/lib/X11/libXw.a

/usr/lib/X11 -> /X11/lib

/X11 -> /usr/local/X11R4

/usr/local/X11R4/lib/libXw.a



$ sl /bin/rnews



/bin -> /usr/bin

/usr/bin/rnews -> /usr/lib/news/rnews

/usr/lib/news -> ../local/lib/news

     local/lib/news/rnews -> inews

                    inews

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you have a group of files whose names end with .new and you want to rename them to end with .old, you might try something like the following:

% mv *.new *.old 

               Wrong!

However, this won't work because the shell can't match *.old and because the mv command just doesn't work that way. Here's one way to do it that will work with most shells:

-d Section 8.5, $..$..\1 Section 34.11

$ ls -d *.new | sed "s/\(.*\)\.new$/mv '&' '\1.old'/" | sh

% ls -d *.new | sed 's/\(.*\)\.new$/mv "&" "\1.old"/' | sh

That outputs a series of mv commands, one per file, and pipes them to a shell (Section 3.4). The quotes help make sure that special characters Section 27.17) aren't touched by the shell — this isn't always needed, but it's a good idea if you aren't sure what files you'll be renaming. Single quotes around the filenames are "strongest"; we use them in the Bourne-type shell version. Unfortunately, csh and tcsh don't allow $ inside double quotes unless it's the start of a shell variable name. So the C shell version puts double quotes around the filenames — but the Bourne shell version can use the "stronger" single quotes, like this:

mv 'afile.new' 'afile.old'

mv 'bfile.new' 'bfile.old'

   ...

To copy, change mv to cp. For safety, use mv -i or cp -i if your versions have the -i options Section 14.15). Using sh -v Section 27.15) will show the commands as the shell executes them.

This method works for any Unix command that takes a pair of filenames. For instance, to compare a set of files in the current directory with the original files in the /usr/local/src directory, use diff :

% ls -d *.c *.h | sed 's@.*@diff -c & /usr/local/src/&@' | sh

Note that diff -r does let you compare entire directories, but you need a trick like this to only compare some of the files.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 10.9 shows how to rename a set of files, e.g., changing *.new to *.old. Here's a different way, done from inside vi. This gives you a chance to review and edit the commands before you run them. Here are the steps:

&& Section 34.10, $ Section 32.5

% vi                

               Start vi without a filename

               :r !ls *.new        

               Read in the list of files, one filename per line

               :%s/.*/mv -i &&/    

               Make mv command lines

               :%s/new$/old/       

               Change second filenames; ready to review

               :w !sh              

               Run commands by writing them to a shell

               :q!                 

               Quit vi without saving

If you've made your own version of ls that changes its output format, that can cause trouble here. If your version gives more than a plain list of filenames in a column, use!/bin/ls instead of just !ls.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I couldn't resist throwing my hat into this ring. I can imagine an unsophisticated user who might not trust himself to replace one pattern with another, but doesn't want to repeat a long list of mv -i commands. (The -i option will prompt if a new name would overwrite an existing file.) Here's a simple script (Section 1.8) that takes a list of filenames (perhaps provided by wildcards) as input and prompts the user for a new name for each file:

#!/bin/sh

# Usage: newname files

for x

do

    echo -n "old name is $x, new name is: "

    read newname

    mv -i "$x" "$newname"

done

For example:

% touch junk1 junk2 junk3

% newname junk*

old name is junk1, new name is: test1

mv: overwrite test1 with junk1? y

old name is junk2, new name is: test2

old name is junk3, new name is: test3

In the first case, test1 already existed, so mv -i prompted.

This script is very simple; I just thought I'd use it to demonstrate that there's more than one way to do it, even if you aren't using Perl.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

cp has a -r (recursive) flag, which copies all the files in a directory tree — that is, all the files in a directory and its subdirectories.

One of our Unix systems has a cp without a -r option. But it also has an rcp ( Section 1.21) command that does have -r. rcp can copy to any machine, not just remote machines. When I need cp -r on that host, I use rcp -r.

cp -r can be used in two ways. The first is much like normal copies; provide a list of files to copy and an existing directory into which to copy them. The -r option just means that source directories will be copied as well as normal files. The second allows you to copy a single directory to another location.

Here's how to do the copy shown in Figure 10-1. This copies the directory /home/jane, with all its files and subdirectories, and creates a subdirectory named jane in the current directory (.) (Section 1.16):
```
% cd /work/bkup

% cp -r /home/jane .

                  
```
How can you copy the contents of the subdirectory called data and all its files (but not the subdirectory itself) into a duplicate directory named data.bak? First make sure that the destination directory doesn't exist. That's because if the last argument to cp is a directory that already exists, the source directory will be copied to a subdirectory of the destination directory (i.e., it will become data.bak/data rather than just data.bak):
```
% cd /home/jane

% cp -r data data.bak

                  
```
Use this to copy the subdirectories Aug and Sep and their files from the directory /home/jim/calendar into the current directory (.):
[..]* Section 33.2
```
% cp -r /home/jim/calendar/[AS]* .

                  
```
In many shells, if you wanted the Oct directory too, but not the file named Output, you can copy just the directories by using the handy curly brace operators (Section 28.4):
```
% cp -r /home/jim/calendar/{Aug,Sep,Oct} .

                  
```

Some gotchas:

Symbolic and hard links (Section 10.4) are copied as files. That can be a good thing; if a symbolic link were not turned into a file along the way, the new symbolic link would point to the wrong place. It can be bad if the link pointed to a really big file; the copy can take up a lot of disk space that you didn't expect. (In Figure 10-1, notice that the symbolic link in

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The tar (Section 39.2) command isn't just for tape archives. It can copy files from disk to disk, too. And even if your computer has cp -r (Section 10.12), there are advantages to using tar.

The obvious way to copy directories with tar is to write them onto a tape archive with relative pathnames — then read back the tape and write it somewhere else on the disk. But tar can also write to a Unix pipe — and read from a pipe. This looks like:

% reading-tar 

               | 

               writing-tar

with one trick: the writing-tar process has a different current directory (Section 24.3, Section 24.4) (the place where you want the copy made) than the reading-tar . To do that, run the writing-tar in a subshell (Section 43.7), or if your tar supports it, use the -C option.

The argument(s) to the reading-tar can be directories or files. Just be sure to use relative pathnames (Section 31.2) that don't start with a slash — otherwise, the writing-tar may write the copies in the same place from where the originals came!

"How about an example," you ask? Figure 10-1 has one. It copies from the directory /home/jane, with all its files and subdirectories. The copy is made in the directory /work/bkup/jane:

% mkdir /work/bkup/jane

% cd /home/jane

% tar cf - . | (cd /work/bkup/jane && tar xvf -)

Or, if you want to use -C:

% tar cf - . | tar xvf - -C /work/bkup/jane

In the subshell version, the && operator (Section 35.14) tells the shell to start

tar xvf

-

only if the previous command (the cd) succeeded. That prevents tar writing files into the same directory from which it's reading — if the destination directory isn't accessible or you flub its pathname. Also, don't use the v (verbose) option in both tars unless you want to see doubled output; one or the other is plenty. I usually put it in the writing-tar to see write progress, as that's more interesting to me than how far ahead the system has cached the read for me.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: diff

The diff command displays different versions of lines that are found when comparing two files. It prints a message that uses ed-like notation (a for append, c for change, and d for delete) to describe how a set of lines has changed. The lines themselves follow this output. The < character precedes lines from the first file and > precedes lines from the second file.

Let's create an example to explain the output produced by diff. Look at the contents of three sample files:

test1	test2	test3
apples	apples	oranges
oranges	oranges	walnuts
walnuts	grapes	chestnuts

When you run diff on test1 and test2, the following output is produced:

$ diff test1 test2

3c3

< walnuts

--

> grapes

The diff command displays the only line that differs between the two files. To understand the report, remember that diff is prescriptive, describing what changes need to be made to the first file to make it the same as the second file. This report specifies that only the third line is affected, exchanging walnuts for grapes. This is more apparent if you use the -e option, which produces an editing script that can be submitted to ed , the Unix line editor. (You must redirect standard output (Section 43.1) to capture this script in a file.)

$ diff -e test1 test2

3c

grapes

.

This script, if run on test1, will bring

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: diff

The diff command displays different versions of lines that are found when comparing two files. It prints a message that uses ed-like notation (a for append, c for change, and d for delete) to describe how a set of lines has changed. The lines themselves follow this output. The < character precedes lines from the first file and > precedes lines from the second file.

Let's create an example to explain the output produced by diff. Look at the contents of three sample files:

test1	test2	test3
apples	apples	oranges
oranges	oranges	walnuts
walnuts	grapes	chestnuts

When you run diff on test1 and test2, the following output is produced:

$ diff test1 test2

3c3

< walnuts

--

> grapes

The diff command displays the only line that differs between the two files. To understand the report, remember that diff is prescriptive, describing what changes need to be made to the first file to make it the same as the second file. This report specifies that only the third line is affected, exchanging walnuts for grapes. This is more apparent if you use the -e option, which produces an editing script that can be submitted to ed , the Unix line editor. (You must redirect standard output (Section 43.1) to capture this script in a file.)

$ diff -e test1 test2

3c

grapes

.

This script, if run on test1, will bring

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can use the diff3 command to look at differences between three files. Here are three sample files, repeated from Section 11.1:

test1	test2	test3
apples	apples	oranges
oranges	oranges	walnuts
walnuts	grapes	chestnuts

For each set of differences, diff3 displays a row of equal signs (====) followed by 1, 2, or 3, indicating which file is different; if no number is specified, then all three files differ. Then, using ed-like notation (Section 11.1), the differences are described for each file:

$ diff3 test1 test2 test3

====3

1:1c

2:1c

  apples

3:0a

====

1:3c

  walnuts

2:3c

  grapes

3:2,3c

  walnuts

  chestnuts

With the output of diff3, it is easy to keep track of which file is which; however, the prescription given is a little harder to decipher. To bring these files into agreement, the first range of text (after ====3) shows that you would have to add apples at the beginning of the third file (3:0a). The second range tells you to change line 3 of the second file to line 3 of the first file — change lines 2 and 3 of the third file, effectively dropping the last line.

The diff3 command also has a -e option for creating an editing script for ed. It doesn't work quite the way you might think. Basically, it creates a script for building the first file from the second and third files.

$ diff3 -e test1 test2 test3

3c

walnuts

chestnuts

.

1d

.

w

q

If you reverse the second and third files, a different script is produced:

$ diff3 -e test1 test3 test2

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The diff examples in Section 11.1 and Section 11.2 show compact formats with just the differences between the two files. But, in many cases, context diff listings are more useful. Context diffs show the changed lines and the lines around them. (This can be a headache if you're trying to read the listing on a terminal and there are many changed lines fairly close to one another: the context will make a huge "before" section, with the "after" section several screenfuls ahead. In that case, the more compact diff formats can be useful.) A related format, unified diff, shows context but doesn't take as much space.

Go to http://examples.oreilly.com/upt3 for more information on: rcs

The rcsdiff command shows differences between revisions in an RCS (Section 39.5) file (and will only be available if you have RCS installed). We'll use it here instead of diff — but the concepts are the same. Incidentally, these examples would also work with cvs diff ( Section 39.7), if you have CVS installed.

The -c option shows before-and-after versions of each changed section of a file. By itself, -c shows three lines above and below each change. Here's an example of a C file before and after some edits; the -c2 option shows two lines of context. The -u option shows changed lines next to each other, not in separate before-and-after sections. Again, an option like -u2 shows two lines of context around a change instead of the default three lines.

Start of a listing

A diff -c listing starts with the two filenames and their last-modified dates ("timestamps"). The first filename (here, atcat.c revision 1.1) has three asterisks (***) before it; the second name (atcat.c revision 1.2) has three dashes (---). These markers identify the two files in the difference listings below:

*** atcat.c 1987/09/19 12:00:44 1.1

--- atcat.c 1987/09/19 12:08:41 1.2

A diff -u listing also starts with the two filenames and their last-modified dates ("timestamps"). The first filename (here,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

After you've used diff for a while, the output is easy to read. Sometimes, though, it's just easier to see two files side-by-side. The sdiff command does that. Between the files, it prints < to point to lines that are only in the first file, > for lines only in the second file, and | for lines that are in both, but different. By default, sdiff shows all the lines in both files. Here's a fairly bogus example that compares two files that contain the output of who (Section 2.8) at different times:

$ sdiff -w75 who1 who2

jake     vt01      Sep 10 10:37      jake     vt01      Sep 10 10:37

uunmv    ttyi1i    Sep 16 11:43   <

jerry    ttyi1j    Sep 15 22:38      jerry    ttyi1j    Sep 15 22:38

jake     ttyp1     Sep  9 14:55      jake     ttyp1     Sep  9 14:55

jake     ttyp2     Sep  9 15:19   |  ellen    ttyp2     Sep 16 12:07

                                  >  carolo   ttyp5     Sep 16 13:03

alison   ttyp8     Sep  9 12:49      alison   ttyp8     Sep  9 12:49

To see only lines that are different, use -s (silent):

$ sdiff -s -w75 who1 who2

2d1

uunmv    ttyi1i    Sep 16 11:43   <

5c4,5

jake     ttyp2     Sep  9 15:19   |  ellen    ttyp2     Sep 16 12:07

                                  >  carolo   ttyp5     Sep 16 13:03

The output lines are usually 130 characters long. That's too long for 80-column-wide screens; if you can put your terminal in 132-column mode or stretch your window, fine. If you can't, use the -w option to set a narrower width, like -w80 for 80-column lines; sdiff will show the first 37 characters from each line (it doesn't write quite all 80 columns). If you can set your printer to compressed type or have a very wide window, use an option like -w170 to get all of each line.

Section 11.5 explains a very useful feature of sdiff: building one file interactively from two files you compare.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One problem you might be tempted to tackle with diff3 (Section 11.2) is sorting out the mess resulting if two people make copies of the same file, and then make changes to their copies. You often find that one version has some things right and another version has other things right. What if you wanted to compile a single version of this document that reflects the changes made to each copy? You want to select which version is correct for each set of differences. An effective way to do this would be to use sdiff (Section 11.4). (Of course, the best thing to do is to prevent the problem in the first place, by using RCS or CVS (Section 39.4).)

One of the most powerful uses of sdiff is to build an output file by choosing between different versions of two files interactively. To do this, specify the -o option and the name of an output file to be created. The sdiff command then displays a % prompt after each set of differences.

You can compare the different versions and select the one that will be sent to the output file. Some of the possible responses are l to choose the left column, r to choose the right column, and q to exit the program.

—TOR and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The diff (Section 11.1) utility adds extra characters (>, <, +, and so on) to the beginning of lines. That can cause you real grief with tabstops because the extra characters added by diff can shift lines enough to make the indentation look wrong. The diff -t option expands TABs to 8-character tabstops and solves the problem.

If you use nonstandard tabstops, though, piping diff's output through expand or pr -e doesn't help because diff has already added the extra characters.

The best answers I've seen are the

<(

)

process-substitution operator and the ! (exclamation point) script. You can expand TABs before diff sees them. For example, to show the differences between two files with 4-column tabstops:

$ diff <(expand -4 afile) <(expand -4 bfile)    

               process substitution

% diff `! expand -4 afile` `! expand -4 bfile`  

               other shells

Of course, nonstandard tabstops cause lots more problems than just with diff. If you can, you're better off using 8-space TABs and using spaces instead of tabs for indentation.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

cmp is another program for comparing files. It's a lot simpler than diff ( Section 11.1); it tells you whether the files are equivalent and the byte offset at which the first difference occurs. You don't get a detailed analysis of where the two files differ. For this reason, cmp is often faster, particularly when you're comparing ASCII files: it doesn't have to generate a long report summarizing the differences. If all you want to know is whether two files are different, it's the right tool for the job.

It's worth noting, though, that cmp isn't always faster. Some versions of diff make some simple checks first, such as comparing file length. If two binary files have different lengths, they are obviously different; some diff implementations will tell you so without doing any further processing.

Both diff and cmp return an exit status ( Section 35.12) that shows what they found:

Exit status	Meaning
0	The files were the same.
1	The files differed.
2	An error occurred.

Within a shell script, the exit status from diff and cmp is often more important than their actual output.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The comm command can tell you what information is common to two lists and what information appears uniquely in one list or the other. For example, let's say you're compiling information on the favorite movies of critics Ebert and Roeper. The movies are listed in separate files (and must be sorted (Section 22.1)). For the sake of illustration, assume each list is short:

% cat roeper

Citizen Kane

Halloween VI

Ninja III

Rambo II

Star Trek V

Zelig

% cat ebert

Cat People

Citizen Kane

My Life as a Dog

Q

Z

Zelig

To compare the favorite movies of your favorite critics, type:

% comm roeper ebert

                  Cat People

                                         Citizen Kane

Halloween VI

                  My Life as a Dog

Ninja III

                  Q

Rambo II

Star Trek V

                  Z

                                         Zelig

Column 1 shows the movies that only Roeper likes; column 2 shows those that only Ebert likes; and column 3 shows the movies that they both like. You can suppress one or more columns of output by specifying that column as a command-line option. For example, to suppress columns 1 and 2 (displaying only the movies both critics like), you would type:

% comm -12 roeper ebert

Citizen Kane

Zelig

As another example, say you've just received a new software release (Release 4), and it's your job to figure out which library functions have been added so that they can be documented along with the old ones. Let's assume you already have a list of the Release 3 functions (r3_list) and a list of the Release 4 functions (r4_list). (If you didn't, you could create them by changing to the directory that has the function manual pages, listing the files with ls, and saving each list to a file.) In the following lists, we've used letters of the alphabet to represent the functions:

% cat r3_list

b

c

d

f

g

h



% cat r4_list

a

b

c

d

e

f

You can now use the comm command to answer several questions you might have:

Which functions are new to Release 4? Answer:

% comm -13 r3_list r4_list    

                     Show 2nd column, which is "Release 4 only"

a

e

Which Release 3 functions have been dropped in Release 4? Answer:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 11.8 didn't show one of my least-favorite comm features. The default output (with text in "columns") confuses me if the lines of output have much text (especially text with spaces). For example, if I'm looking at two who (Section 2.8) listings to compare who was logged on at particular times, the columns in the who output are jumbled:

$ comm who1 who2

                root     tty1     Oct 31 03:13

                jpeek    tty2     Oct 31 03:15

jpeek    pts/0    Oct 31 03:19

                jpeek    pts/1    Oct 31 03:19

                jpeek    pts/2    Oct 31 03:19

ally     pts/4    Oct 31 03:19

                jpeek    pts/3    Oct 31 03:19

        xena     pts/5    Nov  3 08:41

The commer script (see later) filters the comm output through sed. It converts comm's indentation characters (one TAB for lines in "column 2" and two TABs for lines in "column 3") into labels at the start of each output line. The default output looks like this:

$ commer who1 who2

BOTH>root     tty1     Oct 31 03:13

BOTH>jpeek    tty2     Oct 31 03:15

 TWO>jpeek    pts/0    Oct 31 03:19

BOTH>jpeek    pts/1    Oct 31 03:19

BOTH>jpeek    pts/2    Oct 31 03:19

 TWO>ally     pts/4    Oct 31 03:19

BOTH>jpeek    pts/3    Oct 31 03:19

 ONE>xena     pts/5    Nov  3 08:41

With the -i option, the script uses both labels and columns:

$ commer -i who1 who2

BOTH>           root     tty1     Oct 31 03:13

BOTH>           jpeek    tty2     Oct 31 03:15

 TWO>jpeek    pts/0    Oct 31 03:19

BOTH>           jpeek    pts/1    Oct 31 03:19

BOTH>           jpeek    pts/2    Oct 31 03:19

 TWO>ally     pts/4    Oct 31 03:19

BOTH>           jpeek    pts/3    Oct 31 03:19

 ONE>   xena     pts/5    Nov  3 08:41

Go to http://examples.oreilly.com/upt3 for more information on: commer

Here's the script. The sed substitute (s) commands have one or two TABs between the first pair of slashes. Note that the sed script is inside double quotes ("), so the shell can substitute the value of $indent with an ampersand (&) into the sed script if the -i option was used:

#!/bin/sh

# commer - label columns in "comm" output

# Usage: commer [-i] file1 file2

#   -i option indents output lines into columns as "comm" does

#

# Note that script WILL FAIL if any input lines start with a TAB.



case "$1" in

-i) indent='&'; shift ;;

-*|"") echo "Usage: `basename $0` [-i] file1 file2" 1>&2; exit 1 ;;

esac



# In "comm" output, column 1 (lines in file 1) has no leading TAB.

# Column 2 (lines in file 2) has one leading TAB.

# Column 3 (lines in both files) has two leading TABs.

# Search for these tabs and use them to label lines.

# (You could replace ONE and TWO with the filenames $1 and $2)

comm "$1" "$2" |

sed "{

/^              / {s//BOTH>$indent/; b}

/^      / {s// ONE>$indent/; b}

s/^/ TWO>/

}"

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The make program is a Unix facility for describing dependencies among a group of related files, usually ones that are part of the same project. This facility has enjoyed widespread use in software-development projects. Programmers use make to describe how to "make" a program — which source files need to be compiled, which libraries must be included, and which object files need to be linked. By keeping track of these relationships in a single place, individual members of a software-development team can make changes to a single module, run make, and be assured that the program reflects the latest changes made by others on the team.

Only by a leap of the imagination do we group make with the other commands for keeping track of differences between files. However, although it does not compare two versions of the same source file, it can be used to compare versions of a source file and to the formatted output.

Part of what makes Unix a productive environment for text processing is discovering other uses for standard programs. The make utility has many possible applications for a documentation project. One such use is to maintain up-to-date copies of formatted files — which make up a single manual and provide users with a way of obtaining a printed copy of the entire manual without having to know which preprocessors or formatting options (Section 45.13) need to be used.

The basic operation that make performs is to compare two sets of files — for example, formatted and unformatted files — and determine if any members of one set, the unformatted files, are more recent than their counterpart in the other set, the formatted files. This is accomplished by simply comparing the last-modification date (Section 8.2) ("timestamp") of pairs of files. If the unformatted source file has been modified since the formatted file was made, make executes the specified command to "remake" the formatted file.

To use make, you have to write a description file, usually named Makefile (or makefile), that resides in the working directory for the project. The

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Thinking about make will pay off in many ways. One way to get ideas about how to use it is to look at other Makefiles.

One of my favorites is the Makefile for NIS ( Section 1.21) (formerly called YP, or "Yellow Pages"). I like this Makefile because it does something that you'd never think of doing (even though it suits make perfectly): updating a distributed database.

The Makefile is fairly complicated, so I don't want to get into a line-by-line explication; but I will give you a sketch of how it works. Here's the problem: a system administrator updates one or more files (we'll say the passwd file) and wants to get her changes into the NIS database. So you need to check whether the new password file is more recent than the database. Unfortunately, the database isn't represented by a single file, so there's nothing to "check" against. The NIS Makefile handles this situation by creating empty files that serve as timestamps. There's a separate timestamp file for every database that NIS serves. When you type make, make checks every master file against the corresponding timestamp. If a master file is newer than the timestamp, make knows that it has to rebuild part of the database. After rebuilding the database, the Makefile "touches" the timestamp, so that it reflects the time at which the database was built.

The Makefile looks something like this:

passwd: passwd.time

passwd.time:  /etc/master/passwd

         @ lots of commands that rebuild the database

         @ touch passwd.time

         @ more commands to distribute the new database



hosts: hosts.time

hosts.time:  similar stuff

You may never need to write a Makefile this complicated, but you should look for situations in which you can use make profitably. It isn't just for programming.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This chapter talks about the many ways of dumping a file to the screen. Most users know the brute force approach provided by cat (Section 12.2), but there's more to it than that:

Pagers such as more and less (Section 12.3) that give you more control when looking through long files.
Finding out what type of data a file contains before opening it (Section 12.6).
Looking at just the beginning or end of a file (Section 12.8 through Section 12.12).
Numbering lines (Section 12.13).

— TOR

The cat command may well be the first command new users hear about, if only because of its odd name. cat stands for concatenate or, as some would say, catenate. Both words mean the same thing: to connect in a series. The cat command takes its filename arguments and strings their contents together. Essentially, cat takes its input and spits it out again.

cat has many uses, but the four most basic applications are described in the following list. In many ways, they don't illustrate cat so much as they illustrate the shell's output redirection (Section 43.1) mechanism.

First form:
```
% cat 

                     file

% cat 

                     file1 file2 file

                     ...

                  
```
Use this form to display one or more files on the screen. The output doesn't pause when the screen is full. As a result, if your files are more than one screenful long, the output will whiz by without giving you a chance to read it. To read output by screenfuls, use a pager such as less (Section 12.3).
Second form:
```
% cat 

                     file(s) > new_file

                  
```
Use this form when you want to combine several smaller files into one large file. Be sure the destination file does not already exist; otherwise, it will be replaced by the new contents (effectively destroying the original). For example, the command:
```
% cat chap1 chap2 chap3 > book

                  
```
creates a new file, book, composed of three files, one after the other. The three component files still exist as

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This chapter talks about the many ways of dumping a file to the screen. Most users know the brute force approach provided by cat (Section 12.2), but there's more to it than that:

Pagers such as more and less (Section 12.3) that give you more control when looking through long files.
Finding out what type of data a file contains before opening it (Section 12.6).
Looking at just the beginning or end of a file (Section 12.8 through Section 12.12).
Numbering lines (Section 12.13).

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The cat command may well be the first command new users hear about, if only because of its odd name. cat stands for concatenate or, as some would say, catenate. Both words mean the same thing: to connect in a series. The cat command takes its filename arguments and strings their contents together. Essentially, cat takes its input and spits it out again.

cat has many uses, but the four most basic applications are described in the following list. In many ways, they don't illustrate cat so much as they illustrate the shell's output redirection (Section 43.1) mechanism.

First form:
```
% cat 

                     file

% cat 

                     file1 file2 file

                     ...

                  
```
Use this form to display one or more files on the screen. The output doesn't pause when the screen is full. As a result, if your files are more than one screenful long, the output will whiz by without giving you a chance to read it. To read output by screenfuls, use a pager such as less (Section 12.3).
Second form:
```
% cat 

                     file(s) > new_file

                  
```
Use this form when you want to combine several smaller files into one large file. Be sure the destination file does not already exist; otherwise, it will be replaced by the new contents (effectively destroying the original). For example, the command:
```
% cat chap1 chap2 chap3 > book

                  
```
creates a new file, book, composed of three files, one after the other. The three component files still exist as chap1, chap2, and chap3.

Third form:

% cat 

                     file >> existing_file

% cat 

                     files >> existing_file

Use this form to add one or more files to the end of an existing file. For example:

% cat note1 note2 > note_list

% cat note3 >> note_list

Fourth form:
```
% cat > 

                     newfile

                  
```
Use this form as a quick-and-dirty way to create a new file. This is useful when you aren't yet familiar with any of the standard text editors. With this command, everything you type at the keyboard goes into the new file. (You won't be able to back up to a previous line.) To finish your input, enter CTRL-d on a line by itself.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The most popular pager for Unix systems was once the more command, so named because it gave you "one more screen." more is ubiquitous, but also somewhat limited in its capability. The less command (so named because, of course, "less is more!") is more commonly used. less is a full-featured text pager that emulates more but offers an extended set of capabilities.

One particularly important feature of less is that it does not read all of its input before starting, which makes it faster than an editor for large input. less also offers many useful features and is available for almost every operating environment. As an extra bonus, it is installed by default on most free Unixes.

less begins execution by first examining the environment in which it is running. It needs to know some things about the terminal (or window) in which its output will be displayed. Once that's known, less formats the text and displays the first screen's output. The last line of the screen is reserved for user interaction with the program. less will display a colon (:) on the first column of the last line and leave the cursor there. This colon is a command prompt, awaiting command input from the user. Most commands to less are single-character entries, and less will act upon them immediately and without a subsequent carriage return (this is known as cbreak mode). The most basic command to less (and more) is a single space, which instructs the pager to move ahead in the text by one screen. Table 12-1 lists commonly used less commands.

Table 12-1: Commonly used less commands
Command	Description
Space	Scroll forward one screen.
`d`

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Especially if you use an ASCII-based terminal, files can have characters that your terminal can't display. Some characters will lock up your communications software or hardware, make your screen look strange, or cause other weird problems. So if you'd like to look at a file and you aren't sure what's in there, it's not a good idea to just cat the file!

Instead, try cat -v. It shows an ASCII ("printable") representation of unprintable and non-ASCII characters. In fact, although most manual pages don't explain how, you can read the output and see what's in the file. Another utility for displaying nonprintable files is od . I usually use its -c option when I need to look at a file character by character.

Let's look at a file that's almost guaranteed to be unprintable: a directory file. This example is on a standard V7 (Unix Version 7) filesystem. (Unfortunately, some Unix systems won't let you read a directory. If you want to follow along on one of those systems, try a compressed file (Section 15.6) or an executable program from /bin.) A directory usually has some long lines, so it's a good idea to pipe cat's output through fold:

% ls -fa

.

..

comp

% cat -v . | fold -62

M-^?^N.^@^@^@^@^@^@^@^@^@^@^@^@^@>^G..^@^@^@^@^@^@^@^@^@^@^@^@

M-a

comp^@^@^@^@^@^@^@^@^@^@^@^@MassAveFood^@^@^@^@^@hist^@^@^

@^@^@^@^@^@^@^@

% od -c .

0000000 377 016   .  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

0000020   > 007   .   .  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

0000040 341  \n   c   o   m   p  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

0000060  \0  \0   M   a   s   s   A   v   e   F   o   o   d  \0  \0  \0

0000100  \0  \0   h   i   s   t  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

0000120

Each entry in a V7-type directory is 16 bytes long (that's also 16 characters, in the ASCII system). The od -c command starts each line with the number of bytes, in octal, shown since the start of the file. The first line starts at byte 0. The second line starts at byte 20 octal (that's byte 16 in decimal, the way most people count). And so on. Enough about

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The cat -v option (Section 12.4) shows an ASCII representation of unprintable and non-ASCII characters. cat has two options for displaying whitespace in a line. If you use the -t option with -v, TAB characters are shown as ^I. The -e option combined with -v marks the end of each line with a $ character. Some versions of cat don't require the -v with those options. Let's compare a one-line file without and with the -t -e (which may have to be typed separately, by the way; -te won't work on some versions):

% cat afile

This is a one-line file - boring, eh?

% cat -v -t -e afile

ThiS^Hs is^Ia one-line file^I- boring, eh?       $

Although you can't tell it from plain cat, there's a backspace (CTRL-h) before the first s, two TABs that take up only one column of whitespace each, and seven spaces at the end of the line. Knowing this can help you debug problems in printing and displaying files. It's also a help for shell programmers who need to parse or sort the output of other programs.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many different kinds of files live on the typical Unix system: database files, executable files, regular text files, files for applications like StarOffice, tar files, mail messages, directories, font files, and so on.

You often want to check to make sure you have the right "kind" of file before doing something. For example, you'd like to read the file tar. But before typing

more

tar

, you'd like to know whether this file is your set of notes on carbon-based sludge or the tar executable. If you're wrong, the consequences might be unpleasant. Sending the tar executable to your screen might screw up your terminal settings, log you off, or do any number of unpleasant things.

Go to http://examples.oreilly.com/upt3 for more information on: file

The file utility tells you what sort of file something is. It's fairly self-explanatory:

% file /bin/sh

/bin/sh:       sparc demand paged executable

% file 2650

2650:          [nt]roff, tbl, or eqn input text

% file 0001,v

0001,v:        ascii text

% file foo.sh

foo.sh:        shell commands

file is actually quite clever though it isn't always correct — some versions are better than others. It doesn't just tell you if something's binary or text; it looks at the beginning of the file and tries to figure out what it's doing. So, for example, you see that file 2650 is an nroff (Section 45.12) file and foo.sh is a shell script. It isn't quite clever enough to figure out that 0001,v is an RCS (Section 39.5) archive, but it does know that it's a plain ASCII text file.

Many versions of file can be customized to recognize additional file types. The file /etc/magic tells file how to recognize different kinds of files. [My Linux system has the file command from ftp://ftp.astron.com/pub/file/, which uses a multiple-database format. It's updated fairly often to understand new file formats. — JP] It's capable of a lot (and should be capable of even more), but we'll satisfy ourselves with an introductory explanation. Our goal will be to teach file to recognize RCS archives.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Reading output with lots of empty lines can be a waste of screen space. For instance, some versions of man (Section 2.1) show all the blank lines between manual pages. To stop that, read your file or pipe it through cat -s. ( Many versions of less (Section 12.3) and more have a similar -s option.) The -s option replaces multiple blank lines with a single blank line. (If your cat doesn't have -s, see the sed alternative at the end.)

cat -s might not always seem to work. The problem is usually that the "empty" lines have SPACE, TAB, or CTRL-m characters on them. The fix is to let sed "erase" lines with those invisible characters on them:

% sed 's/^[SPACE TAB CTRL-v CTRL-m]*$//' file

                | cat -s

In vi (Section 18.6) and many terminal drivers, the CTRL-v character quotes the CTRL-m (RETURN) so that character doesn't end the current line.

If you don't have cat -s, then sed can do both jobs:

% sed -e 's/^[SPACE TAB CTRL-v CTRL-m]*$//' -e '/./,/^$/!d' file

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Let's say that you want to look at the end of some large file. For example, you've just sent some mail and want to find out whether it was handled correctly. But when you look at your mail logs, you find out that the log file is 30 or 40 KB long, and you don't care about the whole thing — you certainly don't want to page through it until you get to the end. How do you handle this?

The tail command is just what you need in this situation. tail reads its input and discards everything except for the last ten lines (by default). Therefore, if you're pretty sure that the information you want is at the end of the file, you can use tail to get rid of the junk that you don't want. To see just the end of that mail log (in this case, qmail's log):

% tail /var/log/maillog

Feb 19 10:58:45 yyy qmail: 1014141525.474209 delivery 6039: success: did_0+0+1/

Feb 19 10:58:45 yyy qmail: 1014141525.491370 status: local 0/10 remote 0/20

Feb 19 10:58:45 yyy qmail: 1014141525.492211 end msg 111214

Feb 19 11:11:15 yyy qmail: 1014142275.469000 new msg 111214

Feb 19 11:11:15 yyy qmail: 1014142275.469631 info msg 111214: bytes 281 from

<xxx@yyy.zyzzy.com> qp 51342 uid 1000

Feb 19 11:11:15 yyy qmail: 1014142275.562074 starting delivery 6040: msg 111214

to remote xyz@frob.com

Feb 19 11:11:15 yyy qmail: 1014142275.562630 status: local 0/10 remote 1/20

Feb 19 11:11:30 yyy qmail: 1014142290.110546 delivery 6040: success:

64.71.166.115_accepted_message./Remote_host_said:_250_Ok:_queued_as_C0EC73E84D/

Feb 19 11:11:30 yyy qmail: 1014142290.127763 status: local 0/10 remote 0/20

Feb 19 11:11:30 yyy qmail: 1014142290.128381 end msg 111214

For another common example, to see the latest entries from the BSD or Linux kernel ring buffer:

% dmesg | tail

lpt0: <Printer> on ppbus0

lpt0: Interrupt-driven port

ppi0: <Parallel I/O> on ppbus0

IPsec: Initialized Security Association Processing.

ad0: 19569MB <ST320430A> [39761/16/63] at ata0-master UDMA66

afd0: 239MB <IOMEGA ZIP 250 ATAPI> [239/64/32] at ata0-slave using PIO3

acd0: CDROM <ATAPI CDROM> at ata1-master using PIO4

Mounting root from ufs:/dev/ad0s1a

pid 50882 (fetch), uid 0: exited on signal 10 (core dumped)

pid 88041 (smbd), uid 1000 on /usr: file system full

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

What if you need to look at the last 11 lines of the file? The command tail - n shows the final n lines. The command

tail

+

n discards the first n-1 lines, giving you line n and everything that follows it.

You can also tell tail to count the number of characters or the number of 512-byte blocks. To do so, use the -c option (count characters) or the -b option (count blocks). If you want to state explicitly that you're interested in lines, give the -l option.

Your tail probably has a -r option that shows the file in reverse order, starting from the last line.

Many versions of Unix limit the maximum number of lines that tail, especially tail -r, can display.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the best things that you can do with tail is to look at a file as it is growing. For example, I once was debugging a program named totroff that converted a manual from a plain text format to troff. It was rather slow, so that you didn't want to wait until the program finished before looking at the output. But you didn't want to be typing more every 20 seconds either, to find out whether the part of the file that you were debugging had made it through yet. (more quits when you "run out" of file, so it can't really help you look for a part of a file that hasn't been written yet.) The tail -f command solves this problem. For example:

& Section 23.3

% totroff < file.txt > file.ms &

[1] 12345

% tail -f file.ms

.LP

Tail produces output as

the file grows.

   ...

CTRL-c

Now suppose you want to monitor several files at once. Administrators, for example, might want to keep track of several log files, such as /usr/adm/messages, /usr/adm/lpd-errs, UUCP error files, etc. The GNU tail program comes in useful for keeping an eye on several administrative log files at once. But it also comes in useful for nonadministrators.

For example, suppose you want to perform several greps through many files, saving the output in different files. You can then monitor the files using tail -f. For example:

% grep Berkeley ch?? > Berkeley.grep &

% grep BSD ch?? > BSD.grep &

% grep "System V" ch?? > SystemV.grep &

% grep SysV ch?? > SysV.grep &

% tail -f Berkeley.grep BSD.grep SystemV.grep SysV.grep

When new text appears in the files called with tail -f, it also appears on the screen:

==> SysV.grep <==

ch01:using a SysV-based UNIX system, you must



==> Berkeley.grep <==

ch01:at the University of California at Berkeley, where



==> BSD.grep <==

ch03:prefer BSD UNIX systems because they are less likely to

ch04:who use a BSD-based UNIX systems must run the



==> SysV.grep <==

ch04:is a SysV derivative sold by Acme Products Inc.

(When text is written to a new file, the filename is printed surrounded by ==> and <==.)

What's actually happening here?

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I like to keep an xterm window open on my Linux system so I can watch various log files. Although there are fancier log-file-monitoring programs (such as swatch), tail -f (Section 12.10) is perfect for me.

I also run a weekly cron ( Section 25.2) job to rotate log files (rename the files, compress and archive them). When this job runs, the log files suddenly have new names — messages becomes messages.1, for instance — so the system logger starts writing to a different messages file. Then plain tail -f suddenly stops showing the log because it doesn't realize that the same physical file on the disk suddenly has a new name. When this happened, I had to remember to kill and restart tail each Sunday morning . . . until I found the new version of GNU tail, that is.

The GNU tail - -follow option lets you choose how the files you're watching should be followed. By default, GNU tail acts like the standard tail: it opens a file for reading and gets a file descriptor (Section 36.15) number, which it constantly watches for changes. But if that file is renamed and a new file with the old name (and a new inode) takes its place, the file descriptor may point to a file that's not in use anymore.

Go to http://examples.oreilly.com/upt3 for more information on: tail

The GNU tail options, - -follow=name and - -retry, tell it to watch the actual file name, not the open file descriptor. Here's what happens Sunday mornings when I'm using this:

{ } Section 28.4

kludge# tail --follow=name --retry ~jerry/tmp/startx.log \

                 /var/log/{messages,maillog,secure}

               ...lots of log messages...

tail: `/var/log/secure' has been replaced;  following end of new file

tail: `/var/log/maillog' has been replaced;  following end of new file

tail: `/var/log/messages' has been replaced;  following end of new file

Dec 31 04:02:01 kludge syslogd 1.3-3: restart.

Dec 31 04:02:01 kludge syslogd 1.3-3: restart.

Dec 31 04:02:05 kludge anacron[8397]: Updated timestamp for job

`cron.weekly' to 2000-12-31

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

head can be used to print the first few lines of one or more files (the "head" of the file or files). When more than one file is specified, a header is printed at the beginning of each file, and each is listed in succession.

Like tail (Section 12.9), head supports the - n option to control the number of lines displayed and the -c option to print characters/bytes instead of lines. GNU head also supports an extention to -c: -c nk prints the first n kilobytes of the file, and -c nm prints the first n megabytes of the file.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are times when you want to print out a file with the lines numbered — perhaps because you are showing a script or program in documentation and want to refer to individual lines in the course of your discussion.

This is one of the handy things cat can do for you with the -n option.

cat -n precedes each line with some leading spaces, the line number, and a TAB. How many leading spaces? It depends on how high the line numbers go. The line numbers are right-justified at column 6, which means that a 6-digit number will go all the way back to the margin. I only belabor this point in case you're tempted to trim the leading spaces with something like cut (Section 21.14).

Go to http://examples.oreilly.com/upt3 for more information on: nl

If you have a version of cat that doesn't support -n, try nl, the line-numbering program. nl -ba acts like cat -n. By itself, nl numbers only lines with text. The GNU version is on the web site.

You can achieve a similar effect with pr -t -n . (The -t keeps pr from inserting the header and footer (Section 45.6) it normally uses to break its output into pages.) And as long as we're giving you choices, here are five more:

less -N filename

grep -n \^ filename

awk '{print NR,$0}' filename

sed = < filename | sed 'N;s/\n/ /'

ex - '+%#\|q' filename

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

grep is one of Unix's most useful tools. As a result, all users seem to want their own, slightly different version that solves a different piece of the problem. (Maybe this is a problem in itself; there really should be only one grep, as the manpage says.) Three versions of grep come with every Unix system; in addition, there are six or seven freely available versions that we'll mention here, as well as probably dozens of others that you can find kicking around the Net.

Here are the different versions of grep and what they offer. We'll start with the standard versions:

Plain old grep: Great for searching with regular expressions (Section 13.2).
Extended grep (or egrep): Handles extended regular expressions. It is also, arguably, the fastest of the standard greps (Section 13.4).
Fixed grep (or fgrep): So named because it matches fixed strings. It is sometimes inaccurately called "fast grep"; often it is really the slowest of them all. It is useful to search for patterns with literal backslashes, asterisks, and so on that you'd otherwise have to escape somehow. fgrep has the interesting ability to search for multiple strings (Section 13.5).

Of course, on many modern Unixes all three are the same executable, just with slightly different behaviors, and so you may not see dramatic speed differences between them. Now for the freeware versions:

agrep, or "approximate grep": A tool that finds lines that "more or less" match your search string. A very interesting and useful tool, it's part of the glimpse package, which is an indexing and query system for fast searching of huge amounts of text. agrep is introduced in Section 13.6.
Very fast versions of grep, such as GNU grep/egrep/fgrep: Most free Unixes use GNU grep as their main grep.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

grep is one of Unix's most useful tools. As a result, all users seem to want their own, slightly different version that solves a different piece of the problem. (Maybe this is a problem in itself; there really should be only one grep, as the manpage says.) Three versions of grep come with every Unix system; in addition, there are six or seven freely available versions that we'll mention here, as well as probably dozens of others that you can find kicking around the Net.

Here are the different versions of grep and what they offer. We'll start with the standard versions:

Plain old grep: Great for searching with regular expressions (Section 13.2).
Extended grep (or egrep): Handles extended regular expressions. It is also, arguably, the fastest of the standard greps (Section 13.4).
Fixed grep (or fgrep): So named because it matches fixed strings. It is sometimes inaccurately called "fast grep"; often it is really the slowest of them all. It is useful to search for patterns with literal backslashes, asterisks, and so on that you'd otherwise have to escape somehow. fgrep has the interesting ability to search for multiple strings (Section 13.5).

Of course, on many modern Unixes all three are the same executable, just with slightly different behaviors, and so you may not see dramatic speed differences between them. Now for the freeware versions:

agrep, or "approximate grep": A tool that finds lines that "more or less" match your search string. A very interesting and useful tool, it's part of the glimpse package, which is an indexing and query system for fast searching of huge amounts of text. agrep is introduced in Section 13.6.
Very fast versions of grep, such as GNU grep/egrep/fgrep: Most free Unixes use GNU grep as their main grep.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are many well-known benefits provided by grep to the user who doesn't remember what his files contain. Even users of non-Unix systems wish they had a utility with its power to search through a set of files for an arbitrary text pattern (known as a regular expression).

The main function of grep is to look for strings matching a regular expression and print only the lines found. Use grep when you want to look at how a particular word is used in one or more files. For example, here's how to list the lines in the file ch04 that contain either run-time or run time:

".." Section 27.12

$ grep "run[- ]time" ch04

This procedure avoids run-time errors for not-assigned

and a run-time error message is produced.

run-time error message is produced.

program aborts and a run-time error message is produced.

DIMENSION statement in BASIC is executable at run time.

This means that arrays can be redimensioned at run time.

accessible or not open, the program aborts and a run-time

Another use might be to look for a specific HTML tag in a file. The following command will list top-level (<H1> or <h1>) and second-level (<H2> or <h2>) headings that have the starting tag at the beginning (^) of the line:

$ grep "^<[Hh][12]>" ch0[12].html

ch01.html:<h1>Introduction</h1>

ch01.html:<h1>Windows, Screens, and Images</h1>

ch01.html:<h2>The Standard Screen-stdscr</h2>

ch01.html:<h2>Adding Characters</h2>

ch02.html:<H1>Introduction</H1>

ch02.html:<H1>What Is Terminal Independence?</H1>

ch02.html:<H2>Termcap</H2>

ch02.html:<H2>Terminfo</H2>

In effect, it produces a quick outline of the contents of these files.

grep is also often used as a filter (Section 1.5), to select from the output of some other program. For example, you might want to find the process id of your inetd, if you just changed the configuration file and need to HUP inetd to make it reread the configuration file. Using ps ( Section 24.5) and grep together allows you to do this without wading through a bunch of lines of output:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The grep programs have one very handy feature: they can select lines that don't match a pattern just as they can select the lines that do. Simply use the -v option.

I used this most recently when working on this book. We have thousands of separate files under RCS (Section 39.5), and I sometimes forget which ones I've got checked out. Since there's a lot of clutter in the directory and several people working there, a simple ls won't do. There are a series of temporary files created by some of our printing scripts that I don't want to see. All of their filenames consist of one or more x characters: nothing else. So I use a findpt alias to list only the files belonging to me. It's a version of the find. alias described in Section 9.26, with -user tim added to select only my own files and a grep pattern to exclude the temporary files. My findpt alias executes the following command line:

find. | grep -v '^\./xx*$'

The leading ./ matches the start of each line of find. output, and xx* matches one x followed by zero or more xs. I couldn't use the find operators

!

-name

in that case because -name uses shell-like wildcard patterns, and there's no way to say "one or more of the preceding character" (in this case, the character x) with shell wildcards.

Obviously, that's as specific and nonreproducible an example as you're likely to find anywhere! But it's precisely these kinds of special cases that call for a rich vocabulary of tips and tricks. You'll never have to use grep -v for this particular purpose, but you'll find a use for it someday.

[Note that you could use a slightly simpler regular expression by using egrep (Section 13.4), which supports the plus (+) operator to mean "one or more," instead of having to use the basic regular expression character character zero-or-more (xx*). The previous regular expression would then become:

find. | egrep -v '^\./x+$'

The richer regular expression language is the primary advantage of egrep. — DJPH]

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The egrep command is yet another version of grep (Section 13.2), one that extends the syntax of regular expressions. (Versions where grep and egrep are the same allow you to get egrep-like behavior from grep by using the -E option.) A plus sign (+) following a regular expression matches one or more occurrences of the regular expression; a question mark (?) matches zero or one occurrences. In addition, regular expressions can be nested within parentheses:

% egrep "Lab(oratorie)?s" name.list

AT&T Bell Laboratories

AT&T Bell Labs



Symtel Labs of Chicago

Parentheses surround a second regular expression and ? modifies this expression. The nesting helps to eliminate unwanted matches; for instance, the word Labors or oratories would not be matched.

Another special feature of egrep is the vertical bar (|), which serves as an or operator between two expressions. Lines matching either expression are printed, as in the next example:

% egrep "stdscr|curscr" ch03

into the stdscr, a character array.

When stdscr is refreshed, the

stdscr is refreshed.

curscr.

initscr( ) creates two windows: stdscr

and curscr.

Remember to put the expression inside quotation marks to protect the vertical bar from being interpreted by the shell as a pipe symbol. Look at the next example:

% egrep "Alcuin (User|Programmer)('s)? Guide" docguide

Alcuin Programmer's Guide is a thorough

refer to the Alcuin User Guide

Alcuin User's Guide introduces new users to

You can see the flexibility that egrep's syntax can give you, matching either User or Programmer and matching them regardless of whether they had an 's.

Both egrep and fgrep can read search patterns from a file using the -f option (Section 13.5).

— DJPD

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

egrep ( Section 13.4) lets you look for multiple patterns using its grouping and alternation operators (big words for parentheses and a vertical bar). But sometimes, even that isn't enough.

Both egrep and fgrep support a -f option, which allows you to save a list of patterns (fixed strings in the case of fgrep) in a file, one pattern per line, and search for all the items in the list with a single invocation of the program. For example, in writing this book, we've used this feature to check for consistent usage in a list of terms across all articles:

% egrep -f terms *

(To be more accurate, we used rcsegrep (Section 13.7), since the articles are all kept under RCS (Section 39.5), but you get the idea.)

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

agrep is one of the nicer additions to the grep family. It's not only one of the faster greps around; it also has the unique feature of looking for approximate matches. It's also record oriented rather than line oriented. The three most significant features of agrep that are not supported by the grep family are as follows:

The ability to search for approximate patterns, with a user-definable level of accuracy. For example:
```
% agrep -2 homogenos foo

                  
```
will find "homogeneous," as well as any other word that can be obtained from "homogenos" with at most two substitutions, insertions, or deletions.
```
% agrep -B homogenos foo

                  
```
will generate a message of the form:
```
best match has 2 errors, there are 5 matches, output them? (y/n)
```
agrep is record oriented rather than just line oriented; a record is by default a line, but it can be user-defined with the -d option specifying a pattern that will be used as a record delimiter. For example:
```
% agrep -d '^From ' 'pizza' mbox

                  
```
outputs all mail messages (Section 1.21) (delimited by a line beginning with From and a space) in the file mbox that contain the keyword pizza. Another example:
```
% agrep -d '$$' 

                     pattern

                      foo

                  
```
will output all paragraphs (separated by an empty line) that contain pattern.
agrep allows multiple patterns with AND (or OR) logic queries. For example:
```
% agrep -d '^From ' 'burger,pizza' mbox

                  
```
outputs all mail messages containing at least one of the two keywords (, stands for OR).
```
% agrep -d '^From ' 'good;pizza' mbox

                  
```
outputs all mail messages containing both keywords.

Putting these options together, one can write queries such as the following:

% agrep -d '$$' -2 '<CACM>;

               TheAuthor

               ;Curriculum;<198[5-9]>' bib

which outputs all paragraphs referencing articles in CACM between 1985 and 1989 by

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Storing multiple versions of a file in RCS (Section 39.5) saves space. How can you search a lot of those files at once? You could check out all the files, then run grep — but you'll have to remove the files after you're done searching. Or, you could search the RCS files themselves with a command like grep foo RCS/*,v — but that can show you garbage lines from previous revisions, log messages, and other text that isn't in the latest revision of your file. This article has two ways to solve that problem.

The rcsgrep script — and two links to it named rcsegrep and rcsfgrep — run grep , egrep (Section 13.4), and fgrep on all files in the RCS directory. (You can also choose the files to search.)

The script tests its name to decide whether to act like grep, egrep, or fgrep. Then it checks out each file and pipes it to the version of grep you chose. The output looks just like grep's — although, by default, you'll also see the messages from the co command (the -s option silences those messages).

By default, rcsgrep searches the latest revision of every file. With the -a option, rcsgrep will search all revisions of every file, from first to last. This is very handy when you're trying to see what was changed in a particular place and to find which revision(s) have some text that was deleted some time ago. (rcsgrep uses rcsrevs (Section 39.6) to implement -a.)

Some grep options need special handling to work right in the script: -e, -f, and -l. (For instance, -e and -f have an argument after them. The script has to pass both the option and its argument.) The script passes any other options you type to the grep command. Your grep versions may have some other options that need special handling, too. Just edit the script to handle them.

To search an RCS file, rcsgrep and its cousins run several Unix processes: co, grep, sed, and others. Each process takes time to start and run. If your directory has hundreds of RCS files (like our directory for this book does), searching the whole thing can take a lot of time. I could have cut the number of processes by rewriting

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

By default, standard grep utilities show only the lines of text that match the search pattern. Sometimes, though, you need to see the matching line's context: the lines before or after the matching line. The GNU greps (grep, fgrep, and egrep) can do this. There are three context grep options:

The -C option shows two lines of context around each match; you can also give a numeric argument, such as -C 4, to choose how many lines of context (here, four).
The -B option shows context before each match. A numeric argument, such as -B 2 for two lines of context, is required.
The -A option shows context after each match. A numeric argument, such as -A 3 for three lines of context, is required.

Each set of contiguous matching lines is separated by a line of two dashes (--).

Let's look at an example: I'd like to search my system mail log for all messages sent to anyone at oreilly.com. But sendmail doesn't put all information about a message on the to= log line; some info is in the from= line, which is usually the previous line. So I'll search for all "to" lines and add one line of context before each match. I'll also use the -n, which numbers the output lines, to make the context easier to see. This option also puts marker characters after the line number: a line number ends with a colon (:) if this line contains a match, and a dash (-) marks lines before or after a match. Here goes:

# grep -n -B 1 'to=<[^@]*@oreilly\.com>' maillog

7-Nov 12 18:57:42 jpeek sendmail[30148]: SAA30148: from=<jpeek@jpeek.com>...

8:Nov 12 18:57:43 jpeek sendmail[30150]: SAA30148: to=<al@oreilly.com>...

9-Nov 12 22:49:51 jpeek sendmail[1901]: WAA01901: from=<jpeek@jpeek.com>...

10:Nov 12 22:49:51 jpeek sendmail[1901]: WAA01901: to=<wfurby@oreilly.com>...

11:Nov 12 22:50:23 jpeek sendmail[2000]: WAA01901: to=<wfurby@oreilly.com>...

--

25-Nov 13 07:42:38 jpeek sendmail[9408]: HAA09408: from=<jpeek@jpeek.com>...

26:Nov 13 07:42:44 jpeek sendmail[9410]: HAA09408: to=<al@oreilly.com>...

27-Nov 13 08:08:36 jpeek sendmail[10004]: IAA10004: from=<jpeek@jpeek.com>...

28:Nov 13 08:08:37 jpeek sendmail[10006]: IAA10004: to=<wfurby@oreilly.com>...

--

32-Nov 13 11:59:46 jpeek sendmail[14473]: LAA14473: from=<jpeek@jpeek.com>...

33:Nov 13 11:59:47 jpeek sendmail[14475]: LAA14473: to=<al@oreilly.com>...

34-Nov 13 15:34:17 jpeek sendmail[18272]: PAA18272: from=<jpeek@jpeek.com>...

35:Nov 13 15:34:19 jpeek sendmail[18274]: PAA18272: to=<al@oreilly.com>...

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[One weakness of the grep family of programs is that they are line oriented. They read only one line at a time, so they can't find patterns (such as phrases) that are split across two lines. agrep (Section 13.6) can do multiline searches. One advantage of the cgrep script is that it shows how to handle multiple-line patterns in sed and can be adapted for work other than searches. — JP]

Go to http://examples.oreilly.com/upt3 for more information on: cgrep

It may surprise you to learn that a fairly decent context grep (Section 13.8) program can be built using sed. As an example, the following command line:

$ cgrep -10 system main.c

will find all lines containing the word system in the file main.c and show ten additional lines of context above and below each match. (The -context option must be at least one, and it defaults to two lines.) If several matches occur within the same context, the lines are printed as one large "hunk" rather than repeated smaller hunks. Each new block of context is preceded by the line number of the first occurrence in that hunk. This script, which can also search for patterns that span lines:

$ cgrep -3 "awk.*perl"

will find all occurrences of the word "awk" where it is followed by the word "perl" somewhere within the next three lines. The pattern can be any simple regular expression, with one notable exception: because you can match across lines, you should use \n in place of the ^ and $ metacharacters.

[While this is a wonderful example of some neat sed techniques, if this is all you're trying to do, use perl. It has features designed to do exactly this sort of thing very efficiently, and it will be much faster. — DH]

— GU

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You may recall that you can search for lines containing "this" or "that" using the egrep (Section 13.4) | metacharacter:

egrep 'this|that' files

But how do you grep for "this" and "that"? Conventional regular expressions don't support an and operator because it breaks the rule of patterns matching one consecutive string of text. Well, agrep (Section 13.6) is one version of grep that breaks all the rules. If you're lucky enough to have it installed, just use this:

agrep 'cat;dog;bird' files

If you don't have agrep, a common technique is to filter the text through several greps so that only lines containing all the keywords make it through the pipeline intact:

grep cat files | grep dog | grep bird

But can it be done in one command? The closest you can come with grep is this idea:

grep 'cat.*dog.*bird' files

which has two limitations — the words must appear in the given order, and they cannot overlap. (The first limitation can be overcome using egrep 'cat.*dog|dog.*cat', but this trick is not really scalable to more than two terms.)

As usual, the problem can also be solved by moving beyond the grep family to the more powerful tools. Here is how to do a line-by-line and search using sed, awk, or perl:

sed '/cat/!d; /dog/!d; /bird/!d' files

awk '/cat/ && /dog/ && /bird/' files

perl -ne 'print if /cat/ && /dog/ && /bird/' files

Okay, but what if you want to find where all the words occur in the same paragraph? Just turn on paragraph mode by setting RS="" in awk or by giving the -00 option to perl:

awk '/cat/ && /dog/ && /bird/ {print $0 ORS}' RS= files

perl -n00e 'print "$_\n" if /cat/ && /dog/ && /bird/' files

And if you just want a list of the files that contain all the words anywhere in them? Well, perl can easily slurp in entire files if you have the memory and you use the -0 option to set the record separator to something that won't occur in the file (like NUL):

perl -ln0e 'print $ARGV if /cat/ && /dog/ && /bird/'

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're searching a long file to find a particular word or name, or you're running a program like ls -l and you want to filter some lines, here's a quick way to narrow down the search. As an example, say your phone file has 20,000 lines like these:

Smith, Nancy:MFG:50 Park Place:Huntsville:(205)234-5678

and you want to find someone named Nancy. When you see more information, you know you can find which of the Nancys she is:

% grep Nancy phones

               ...150 lines of names...

Use the C shell's history mechanism ( Section 30.2) and sed to cut out lines you don't want. For example, about a third of the Nancys are in Huntsville, and you know she doesn't work there:

% !! | sed -e /Huntsville/d

grep Nancy phones | sed -e /Huntsville/d

...100 lines of names...

The shell shows the command it's executing: the previous command (!!) piped to sed, which deletes lines in the grep output that have the word Huntsville.

Okay. You know Nancy doesn't work in the MFG or SLS groups, so delete those lines, too:

% !! -e /MFG/d -e /SLS/d

grep Nancy phones | sed -e /Huntsville/d -e /MFG/d -e /SLS/d

...20 lines of names...

Keep using !! to repeat the previous command line, and keep adding more sed expressions until the list gets short enough. The same thing works for other commands. When you're hunting for errors in a BSDish system log, for example, and you want to skip lines from named and sudo, use the following:

% cat /var/log/messages | sed -e /named/d -e /sudo/d

...

If the matching pattern has anything but letters and numbers in it, you'll have to understand shell quoting (Section 27.12) and sed regular expressions. Most times, though, this quick-and-dirty method works just fine.

[Yes, you can do the exact same thing with multiple grep -v ( Section 13.3) commands, but using sed like this allows multiple matches with only one execution of sed. grep -v requires a new grep process for each condition. — DH]

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This may be the simplest tip in the book, but it's something that doesn't occur to lots of users.

Some versions of egrep don't support the -i option, which requests case-insensitive searches. I find that case-insensitive searches are absolutely essential, particularly to writers. You never know whether any particular word will be capitalized.

To fake a case-insensitive search with egrep, just eliminate any letters that might be uppercase. Instead of searching for Example, just search for xample. If the letter that might be capitalized occurs in the middle of a phrase, you can replace the missing letter with a "dot" (single character) wildcard, rather than omitting it.

Sure, you could do this the "right way" with a command like:

% egrep '[eE]xample' *

but our shortcut is easier.

This tip obviously isn't limited to egrep; it applies to any utility that only implements case-sensitive searches, like more .

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's an idea for finding lines that have a given character in a column. Use the following simple awk (Section 20.10) command:

% awk 'substr($0,

               n

               ,1) == "

               c

               "' 

               filename

where c is the character you're searching for, and n is the column you care about.

Where would you do this? If you're processing a file with strict formatting, this might be useful; for example, you might have a telephone list with a # in column 2 for "audio" telephone numbers, $ for dialup modems, and % for fax machines. A script for looking up phone numbers might use an awk command like this to prevent you from mistakenly talking to a fax machine.

If your data has any TAB characters, the columns might not be where you expect. In that case, use expand on the file, then pipe it to awk.

—JP and ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Every so often, someone has designed a new, faster grep-type program. Public- domain software archives have more than a few of them. One of the fastest search programs has been around for years: look. It uses a binary search method that's very fast. But look won't solve all your problems: it works only on files that have been sorted (Section 22.1). If you have a big file or database that can be sorted, searching it with look will save a lot of time. For example, to search for all lines that start with Alpha:

% look Alpha 

               filename

Alpha particle

Alphanumeric

Go to http://examples.oreilly.com/upt3 for more information on: look

The look program can also be used to check the spelling of a word or find a related word; see Section 16.3. If you don't have look installed on your system, you can get it from the Unix Power Tools web site.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you try to read binaries on your screen with cat -v (Section 12.4), you'll see a lot of nonprintable characters. Buried in there somewhere, though, are words and strings of characters that might make some sense. For example, if the code is copyrighted, you can usually find that information in the binary. The pathnames of special files read by the program will probably show up. If you're trying to figure out which program printed an error message, use strings on the binaries and look for the error. Some versions of strings do a better job of getting just the useful information; others may write a lot of junk, too. But what the heck? — pipe the output to a pager (Section 12.3) or grep (Section 13.2), redirect it to a file, and ignore the stuff you don't want.

Here's a (shortened) example on FreeBSD:

% strings /usr/bin/write

/usr/libexec/ld-elf.so.1

FreeBSD

libc.so.4

strcpy

...

@(#) Copyright (c) 1989, 1993

        The Regents of the University of California.  All rights reserved.

$FreeBSD: src/usr.bin/write/write.c,v 1.12 1999/08/28 01:07:48 peter Exp $

can't find your tty

can't find your tty's name

you have write permission turned off

/dev/

%s is not logged in on %s

%s has messages disabled on %s

usage: write user [tty]

/var/run/utmp

utmp

%s is not logged in

%s has messages disabled

%s is logged in more than once; writing to %s

%s%s

Message from %s@%s on %s at %s ...

The eighth line ($FreeBSD: ... $) comes from RCS (Section 39.5) — you can see the version number, the date the code was last modified or released, and so on. The %s is a special pattern that the printf(3) function will replace with values like the username, hostname, and time.

By default, strings doesn't search all of a binary file: it only reads the initialized and loaded sections. The - (dash) option tells strings to search all of the file. Another useful option is - n, where n is the minimum-length string to print. Setting a higher limit will cut the "noise," but you might also lose what you're looking for.

The od command with its option -s

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Do you ever grep for a word, and when lines scroll down your screen, it's hard to find the word on each line? For example, suppose I'm looking for any mail messages I've saved that say anything about the perl programming language. But when I grep the file, most of it seems useless:

% grep perl ~/Mail/save

> and some of it wouldn't compile properly.  I wonder if

Subject: install script, for perl scripts

 perl itself is installed?

> run but dies with a read error because it isn't properly

> if I can get it installed properly on another machine I

> run but dies with a read error because it isn't properly

> if I can get it installed properly on another machine I

Well, as described on its own manual page, here's a program that's "trivial, but cute." hgrep runs a grep and highlights the string being searched for, to make it easier for us to find what we're looking for.

% hgrep perl ~/Mail/save

> and some of it wouldn't compile properly.  I wonder if

Subject: install script, for perl scripts

 perl itself is installed?

> run but dies with a read error because it isn't properly

> if I can get it installed properly on another machine I

> run but dies with a read error because it isn't properly

> if I can get it installed properly on another machine I

And now we know why the output looked useless: because most of it is! Luckily, hgrep is just a frontend; it simply passes all its arguments to grep. So hgrep necessarily accepts all of grep's options, and I can just use the -w option to pare the output down to what I want:

% hgrep -w perl ~/Mail/save

Subject: install script, for perl scripts

 perl itself is installed?

The less ( Section 12.3) pager also automatically highlights matched patterns as you search.

— LM

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As a computer user, you spend lots of time creating files. Just as the necessary counterpart of life is death, the other side of file creation is deletion. If you never delete any files, you soon have a computer's equivalent of a population explosion: your disks get full, and you must either spend money (buy and install more disk drives) or figure out which files you don't really need.

In this chapter, we'll talk about ways to get rid of files: how to do it safely, how to get rid of files that don't want to die, and how to find "stale" files — or unused files that have been around for a long time. "Safe" deletion is a particularly interesting topic, because Unix's rm command is extreme: once you delete a file, it's gone permanently. There are several solutions for working around this problem, letting you (possibly) reclaim files from the dead.

— ML

The ability to mumble about inodes is the key to social success at a Unix gurus' cocktail party. This may not seem attractive to you, but sooner or later you will need to know what an inode is.

Seriously, inodes are an important part of the Unix filesystem. You don't need to worry about them most of the time, but it does help to know what they are.

An inode is a data structure on the disk that describes a file. It holds most of the important information about the file, including the on-disk address of the file's data blocks (the part of the file that you care about). Each inode has its own identifying number, called an i-number.

You really don't care about where a file is physically located on a disk. You usually don't care about the i-number — unless you're trying to find the links ( Section 9.24, Section 10.3) to a file. But you do care about the following information, all of which is stored in a file's inode:

The file's ownership: The user and the group that own the file
The file's access mode (Section 1.17, Section 50.2): Whether various users and groups are allowed to read, write, or execute the file

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As a computer user, you spend lots of time creating files. Just as the necessary counterpart of life is death, the other side of file creation is deletion. If you never delete any files, you soon have a computer's equivalent of a population explosion: your disks get full, and you must either spend money (buy and install more disk drives) or figure out which files you don't really need.

In this chapter, we'll talk about ways to get rid of files: how to do it safely, how to get rid of files that don't want to die, and how to find "stale" files — or unused files that have been around for a long time. "Safe" deletion is a particularly interesting topic, because Unix's rm command is extreme: once you delete a file, it's gone permanently. There are several solutions for working around this problem, letting you (possibly) reclaim files from the dead.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The ability to mumble about inodes is the key to social success at a Unix gurus' cocktail party. This may not seem attractive to you, but sooner or later you will need to know what an inode is.

Seriously, inodes are an important part of the Unix filesystem. You don't need to worry about them most of the time, but it does help to know what they are.

An inode is a data structure on the disk that describes a file. It holds most of the important information about the file, including the on-disk address of the file's data blocks (the part of the file that you care about). Each inode has its own identifying number, called an i-number.

You really don't care about where a file is physically located on a disk. You usually don't care about the i-number — unless you're trying to find the links ( Section 9.24, Section 10.3) to a file. But you do care about the following information, all of which is stored in a file's inode:

The file's ownership: The user and the group that own the file
The file's access mode (Section 1.17, Section 50.2): Whether various users and groups are allowed to read, write, or execute the file
The file's timestamps (Section 8.2): When the file itself was last modified, when the file was last accessed, and when the inode was last modified
The file's type: Whether the file is a regular file, a special file, or some other kind of abstraction masquerading (Section 1.19) as a file

Each filesystem has a set number of inodes that are created when the filesystem is first created (usually when the disk is first initialized). This number is therefore the maximum number of files that the filesystem can hold. It cannot be changed without reinitializing the filesystem, which destroys all the data that the filesystem holds. It is possible, though rare, for a filesystem to run out of inodes, just as it is possible to run out of storage space — this can happen on filesystems with many, many small files.

The ls -l (Section 50.2) command shows much of this information. The

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Under Unix, you use the rm command to delete files. The command is simple enough; you just type rm followed by a list of files. If anything, rm is too simple. It's easy to delete more than you want, and once something is gone, it's permanently gone. There are a few hacks that make rm somewhat safer, and we'll get to those momentarily. But first, here's a quick look at some of the dangers.

To understand why it's impossible to reclaim deleted files, you need to know a bit about how the Unix filesystem works. The system contains a "free list," which is a list of disk blocks that aren't used. When you delete a file, its directory entry (which gives it its name) is removed. If there are no more links (Section 10.3) to the file (i.e., if the file only had one name), its inode (Section 14.2) is added to the list of free inodes, and its datablocks are added to the free list.

Well, why can't you get the file back from the free list? After all, there are DOS utilities that can reclaim deleted files by doing something similar. Remember, though, Unix is a multitasking operating system. Even if you think your system is a single-user system, there are a lot of things going on "behind your back": daemons are writing to log files, handling network connections, processing electronic mail, and so on. You could theoretically reclaim a file if you could "freeze" the filesystem the instant your file was deleted — but that's not possible. With Unix, everything is always active. By the time you realize you made a mistake, your file's data blocks may well have been reused for something else.

When you're deleting files, it's important to use wildcards carefully. Simple typing errors can have disastrous consequences. Let's say you want to delete all your object (.o) files. You want to type:

% rm *.o

But because of a nervous twitch, you add an extra space and type:

% rm * .o

It looks right, and you might not even notice the error. But before you know it, all the files in the current directory will be gone, irretrievably.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's a summary of ways to protect yourself from accidentally deleting files:

Use rm -i, possibly as an alias (Section 14.8).
Make rm -i less painful (Section 14.7).
Write a "delete" script that moves "deleted" files to a temporary directory (Section 14.9).
tcsh has an rmstar variable that makes the shell ask for confirmation when you type something like rm *. In zsh, this protection is automatic unless you set the RM_STAR_SILENT shell option to stop it.
Use revision control (Section 39.4).
Make your own backups, as explained in Section 38.3.
Prevent deletion (or renaming or creating) of files by making the directory (not necessarily the files in it!) unwritable (Section 50.2).

If you want to delete with wild abandon, use rm -f (Section 14.10).

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some commands — like rm -i, find -ok, and so on — ask users to answer a "do it or not?" question from the keyboard. For example, you might have a file-deleting program or alias named del that asks before deleting each file:

% del *

Remove file1? y

Remove file2? y

   ...

If you answer y, then the file will be deleted.

What if you want to run a command that will ask you 200 questions and you want to answer y to all of them, but you don't want to type all those ys from the keyboard? Pipe the output of yes to the command; it will answer y for you:

% yes | del *

Remove file1?

Remove file2?

   ...

If you want to answer n to all the questions, you can do:

% yes n | del *

Not all Unix commands read their standard input for answers to prompts. If a command opens your terminal (/dev/tty (Section 36.15)) directly to read your answer, yes won't work. Try expect (Section 28.18) instead.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most people use rm -i for safety: so they're always asked for confirmation before removing a particular file. Mike Loukides told me about another way he uses rm -i. When he has several files to remove, but the wildcards (Section 1.13) would be too painful to type with a plain rm, Mike gives rm -i a bigger list of filenames and answers "n" to filenames he doesn't want deleted. For instance:

% ls

aberrant    abhorred    abnormal     abominate   acerbic

aberrate    abhorrent   abominable   absurd      acrimonious

    ...

% rm -i ab*

rm: remove aberrant (y/n)? y

rm: remove aberrate (y/n)? n

rm: remove abhorred (y/n)? y

rm: remove abhorrent (y/n)? n

    ...

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The rm -i command asks you about each file, separately. The method in this article can give you the safety without the hassle of typing y as much.

Another approach, which I recommend, is that you create a new script or alias, and use that alias whenever you delete files. Call the alias del or Rm, for instance. This way, if you ever execute your special delete command when it doesn't exist, no harm is done — you just get an error. If you get into this habit, you can start making your delete script smarter. Here is one that asks you about each file if there are three or fewer files specified. For more than three files, it displays them all and asks you once if you wish to delete them all:

#!/bin/sh

case $# in

0)     echo "`basename $0`: you didn't say which file(s) to delete"; exit 1;;

[123]) /bin/rm -i "$@" ;;

*)     echo "$*"

       echo do you want to delete these files\?

       read a

       case "$a" in

       [yY]*) /bin/rm "$@" ;;

       esac

       ;;

esac

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Using noclobber (Section 43.6) and read-only files only protects you from a few occasional mistakes. A potentially catastrophic error is typing:

% rm * .o

instead of:

% rm *.o

In the blink of an eye, all of your files would be gone. A simple, yet effective, preventive measure is to create a file called -i in the particular directory in which you want extra protection:

./- Section 14.13

% touch ./-i

In this case, the * is expanded to match all of the filenames in the directory. Because the file -i is alphabetically listed before any file except those that start with one of the characters !, #, $, %, &, ', ( , ), *, +, or ,, the rm command sees the -i file as a command-line argument. When rm is executed with its -i option, files will not be deleted unless you verify the action. This still isn't perfect, though. If you have a file that starts with a comma (,) in the directory, it will come before the file starting with a dash, and rm will not get the -i argument first.

The -i file also won't save you from errors like this:

% rm [a-z]* .o

If lots of users each make a -i file in each of their zillions of subdirectories, that could waste a lot of disk inodes (Section 14.2). It might be better to make one -i file in your home directory and hard link ( Section 15.4) the rest to it, like this:

~ Section 30.11

% cd

% touch ./-i

% cd

                somedir

% ln ~/-i .

   ...

Second, to save disk blocks, make sure the -i file is zero-length — use the touch command, not vi or some other command that puts characters in the file.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

To protect themselves from accidentally deleting files, some users create a " trash" directory somewhere and then write a "safe delete" program that, instead of rming a file, moves it into the trash directory. The implementation can be quite complex, but a simple alias or shell function will do most of what you want:

alias del "mv \!* ~/trash/."

Or, for Bourne-type shells:

del ( ) { mv "$@" $HOME/trash/.; }

Of course, now your deleted files collect in your trash directory, so you have to clean that out from time to time. You can do this either by hand or automatically, via a cron ( Section 25.2) entry like this:

&& Section 35.14, -r Section 14.16

23 2 * * * cd $HOME/trash && rm -rf *

This deletes everything in the trash directory at 2:23 a.m. daily. To restore a file that you deleted, you have to look through your trash directory by hand and put the file back in the right place. That may not be much more pleasant than poking through your garbage to find the tax return you threw out by mistake, but (hopefully) you don't make lots of mistakes.

There are plenty of problems with this approach. Obviously, if you delete two files with the same name in the same day, you're going to lose one of them. A shell script could (presumably) handle this problem, though you'd have to generate a new name for the deleted file. There are also lots of nasty side effects and "gotchas," particularly if you want an rm -r equivalent, if you want this approach to work on a network of workstations, or if you use it to delete files that are shared by a team of users.

Unfortunately, this is precisely the problem. A "safe delete" that isn't really safe may not be worth the effort. Giving people a safety net with holes in it is only good if you can guarantee in advance that they won't land in one of the holes, believing themselves protected. You can patch some of the holes by replacing this simple alias with a shell script; but you can't fix all of them.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The -f option to rm is the extreme opposite of -i. It says, "Just delete the file; don't ask me any questions." The "f" stands (allegedly) for "force," but this isn't quite right. rm -f won't force the deletion of something that you aren't allowed to delete. (To understand what you're allowed to delete, you need to understand directory access permissions (Section 50.2).)

What, then, does rm -f do, and why would you want to use it?

Normally, rm asks you for confirmation if you tell it to delete files to which you don't have write access — you'll get a message like Override protection 444 for foo? (The Unix filesystem allows you to delete read-only files, provided you have write access to the directory.) With -f, these files will be deleted silently.
Normally, rm's exit status ( Section 35.12) is 0 if it succeeded and 1 if it failed to delete the file. With -f, rm's return status is always 0.

I find that I rarely use rm -f on the Unix command line, but I almost always use it within shell scripts. In a shell script, you (probably) don't want to be interrupted by lots of prompts should rm find a bunch of read-only files.

You probably also don't want to be interrupted if rm -f tries to delete files that don't exist because the script never created them. Generally, rm -f keeps quiet about files that don't exist; if the desired end result is for the file to be gone, it not existing in the first place is just as good.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A perennial problem is deleting files that have strange characters (or other oddities) in their names. The next few articles contain some hints for the following:

Deleting files with random control characters in their names (Section 14.12).
Deleting files whose names start with a dash (Section 14.13).
Deleting files with "unprintable" filenames (Section 14.14).
Deleting files by using the inode number (Section 14.15).
Deleting directories and problems that can arise as a result (Section 14.16).

We'll also give hints for these:

Deleting unused (or rarely used) files (Section 14.17).
Deleting all the files in a directory, except for one or two (Section 14.18).

Most tips for deleting files also work for renaming the files (if you want to keep them): just replace the rm command with mv.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Filenames can be hard to handle if their names include control characters or characters that are special to the shell. Here's a directory with three oddball filenames:

% ls

What now

a$file

prog|.c

program.c

When you type those filenames on the command line, the shell interprets the special characters (space, dollar sign, and vertical bar) instead of including them as part of the filename. There are several ways (Section 14.11) to handle this problem. One is with wildcards (Section 33.2). Type a part of the filename without the weird characters, and use a wildcard to match the rest. The shell doesn't scan the filenames for other special characters after it interprets the wildcards, so you're (usually) safe if you can get a wildcard to match. For example, here's how to rename What now to Whatnow, remove a$file, and rename prog|.c to prog.c:

% mv What* Whatnow

% rm -i a*

rm: remove a$file? y

% mv prog?.c prog.c

Filenames with control characters are just another version of the same problem. Use a wildcard to match the part of the name that's troubling you. The real problem with control characters in filenames is that some control characters do weird things to your screen. Once I accidentally got a file with a CTRL-L in its name. Whenever I ran ls, it erased the screen before I could see what the filename was! Section 8.12 explains how, depending on your version of ls, you can use the -q or -b options to spot the offensive file and construct a wildcard expression to rename or delete it. (ls -q is the default on most Unix implementations these days, so you will probably never see this particular problem.)

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sometimes you can slip and create a file whose name starts with a dash (-), like -output or -f. That's a perfectly legal filename. The problem is that Unix command options usually start with a dash. If you try to type that filename on a command line, the command might think you're trying to type a command option.

In almost every case, all you need to do is "hide" the dash from the command. Start the filename with ./ (dot slash). This doesn't change anything as far as the command is concerned; ./ just means "look in the current directory" (Section 1.16). So here's how to remove the file -f:

% rm ./-f

(Most rm commands have a special option for dealing with filenames that start with a dash, but this trick should work on all Unix commands.)

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some versions of Unix have a lot of trouble with eight-bit filenames — that is, filenames that contain non-ASCII characters. The ls -q ( Section 8.12) command shows the nonASCII characters as question marks (?), but usual tricks like rm -i * (Section 14.12) skip right over the file. You can see exactly what the filename is by using ls -b ( Section 8.12):

% ls -q

    ????

afile

bfile

% rm -i *

afile: ? n

bfile: ? n

% ls -b

\t\360\207\005\254

afile

bfile

On older Unixes, the -b option to ls might not be supported, in which case you can use od -c (Section 12.4) to dump the current directory, using its relative pathname . (dot) (Section 1.16), character by character. It's messier, and isn't supported on all Unix platforms, but it's worth a try:

% od -c .

   ...

00.....   \t 360 207 005 254  \0  \0  \0  \0  ...

If you can move all the other files out of the directory, then you'll probably be able to remove the leftover file and directory with rm -rf (Section 14.16, Section 14.10). Moving files and removing the directory is a bad idea, though, if this is an important system directory like /bin. Otherwise, if you use the escaped name ls -b gave you, you might be able to remove it directly by using the system call unlink (2) in Perl. Use the same escape characters in Perl that ls -b displayed. (Or, if you needed to use od -c, find the filename in the od listing of the directory — it will probably end with a series of NUL characters, like

\0 \0

\0

.)

perl -e 'unlink("\t\360\207\005\254");'

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If wildcards don't work (Section 14.12) to remove a file with a strange name, try getting the file's i-number (Section 13.2). Then use find's -inum operator (Section 9.9) to remove the file.

Here's a directory with a weird filename. ls (with its default -q option (Section 8.12) on most versions) shows that the name has three unusual characters. Running ls -i shows each file's i-number. The strange file has i-number 6239. Give the i-number to find, and the file is gone:

% ls

adir      afile     b???file  bfile     cfile     dfile

% ls -i

  6253 adir        6239 b???file    6249 cfile

  9291 afile       6248 bfile       9245 dfile

% find . -inum 6239 -exec rm {} \;

% ls

adir   afile  bfile  cfile  dfile

Instead of deleting the file, I also could have renamed it to newname with the command:

% find . -inum 6239 -exec mv {} newname \;

If the current directory has large subdirectories, you'll probably want to keep find from recursing down into them by using the -maxdepth 1 operator. (finds that don't support -maxdepth can use -prune ( Section 9.25) for speed.)

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

What if you want to get rid of a directory? The standard — and safest — way to do this is to use the Unix rmdir "remove directory" utility:

% rmdir files

The rmdir command often confuses new users. It will only remove a directory if it is completely empty; otherwise, you'll get an error message:

% rmdir files

rmdir: files: Directory not empty

% ls files

%

As in the example, ls will often show that the directory is empty. What's going on?

It's common for editors and other programs to create " invisible" files (files with names beginning with a dot). The ls command normally doesn't list them; if you want to see them, you have to use ls -A (Section 8.9):

% rmdir files

rmdir: files: Directory not empty

% ls -A files

.BAK.textfile2

Here, we see that the directory wasn't empty after all: there's a backup file that was left behind by some editor. You may have used

rm

*

to clean the directory out, but that won't work: rm also ignores files beginning with dots, unless you explicitly tell it to delete them. We really need a wildcard pattern like .??* or .[a-zA-Z0-9]* to catch normal dotfiles without catching the directories . and ..:

% rmdir files

rmdir: files: Directory not empty

% ls -A files

.BAK.textfile2

% rm files/.??*

% rmdir files

%

Other pitfalls might be files whose names consist of nonprinting characters or blank spaces — sometimes these get created by accident or by malice (yes, some people think this is funny). Such files will usually give you "suspicious" ls output (Section 8.11) (like a blank line).

If you don't want to worry about all these special cases, just use rm -r :

% rm -r files

This command removes the directory and everything that's in it, including other directories. A lot of people warn you about it; it's dangerous because it's easy to delete more than you realize. Personally, I use it all the time, and I've never made a mistake. I never bother with rmdir.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sooner or later, a lot of junk collects in your directories: files that you don't really care about and never use. It's possible to write find (Section 9.1) commands that will automatically clean these up. If you want to clean up regularly, you can add some find commands to your crontab file ( Section 25.2).

Basically, all you need to do is write a find command that locates files based on their last access time (-atime (Section 9.5)) and use -ok or -exec (Section 9.9) to delete them. Such a command might look like this:

% find . -atime +60 -ok rm -f {} \;

This locates files that haven't been accessed in the last 60 days, asks if you want to delete the file, and then deletes the file. (If you run it from cron, make sure you use -exec instead of -ok, and make absolutely sure that the find won't delete files that you think are important.)

Of course, you can modify this find command to exclude (or select) files with particular names; for example, the following command deletes old core dumps and GNU Emacs backup files (whose names end in ~), but leaves all others alone:

% find . \( -name core -o -name "*~" \) -atime +60 -ok rm -f {} \;

If you take an automated approach to deleting stale files, watch out for these things:

There are plenty of files (for example, Unix utilities and log files) that should never be removed. Never run any "automatic deletion" script on /usr or / or any other "system" directory.
On some systems, executing a binary executable doesn't update the last access time. Since there's no reason to read these files, you can expect them to get pretty stale, even if they're used often. But you don't want to delete them. If you cook up a complicated enough find command, you should be able to handle this automatically. Something like this should (at least partially) do the trick:
! Section 9.6, -perm Section 9.15
```
% find . -atime +30 ! -perm -111 ... -exec rm {} \;
```

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One problem with Unix: it's not terribly good at "excluding" things. There's no option to rm that says, "Do what you will with everything else, but please don't delete these files." You can sometimes create a wildcard expression (Section 33.2) that does what you want — but sometimes that's a lot of work, or maybe even impossible.

Here's one place where Unix's command substitution ( Section 28.14) operators (backquotes) come to the rescue. You can use ls to list all the files, pipe the output into a grep -v or egrep -v ( Section 13.3) command, and then use backquotes to give the resulting list to rm. Here's what this command would look like:

% rm -i `ls -d *.txt | grep -v '^john\.txt$'`

This command deletes all files whose names end in .txt, except for john.txt. I've probably been more careful than necessary about making sure there aren't any extraneous matches; in most cases, grep -v john would probably suffice. Using ls -d (Section 8.5) makes sure that ls doesn't look into any subdirectories and give you those filenames. The rm -i asks you before removing each file; if you're sure of yourself, omit the -i.

Of course, if you want to exclude two files, you can do that with egrep:

% rm `ls -d *.txt | egrep -v 'john|mary'`

(Don't forget to quote the vertical bar (|), as shown earlier, to prevent the shell from piping egrep's output to mary.)

Another solution is the nom (Section 33.8) script.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Do you run find on your machine every night? Do you know what it has to go through just to find out if a file is three days old and smaller than ten blocks or owned by "fred" or setuid root? This is why I tried to combine all the things we need done for removal of files into one big find script:

Go to http://examples.oreilly.com/upt3 for more information on: cleanup

2>&1 Section 36.16

#! /bin/sh

#

# cleanup - find files that should be removed and clean them

# out of the file system.



find / \(    \( -name '#*'                 -atime +1 \)  \

        -o   \( -name ',*'                 -atime +1 \)  \

        -o   \( -name rogue.sav            -atime +7 \)  \

        -o   \(      \( -name '*.bak'                    \

                     -o -name '*.dvi'                    \

                     -o -name '*.CKP'                    \

                     -o -name '.*.bak'                   \

                     -o -name '.*.CKP' \)  -atime +3 \)  \

        -o   \( -name '.emacs_[0-9]*'      -atime +7 \)  \

        -o   \( -name core                           \)  \

        -o   \( -user guest                -atime +9 \)  \

\) -print -exec rm -f {} \; > /tmp/.cleanup 2>&1

This is an example of using a single find command to search for files with different names and last-access times (see Section 9.5). Doing it all with one find is much faster — and less work for the disk — than running a lot of separate finds. The parentheses group each part of the expression. The neat indentation makes this big thing easier to read. The -print -exec at the end removes each file and also writes the filenames to standard output, where they're collected into a file named /tmp/.cleanup — people can read it to see what files were removed. You should probably be aware that printing the names to /tmp/.cleanup lets everyone see pathnames, such as /home/joe/personal/resume.bak, which some people might consider sensitive. Another thing to be aware of is that this

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many of the techniques in this chapter aren't nearly as applicable as they once were. At the time of this writing, EIDE disks are about a dollar a gigabyte; even fast-wide SCSI isn't that expensive. Often the solution to running low on disk space is just to buy more.

That said, many of these techniques illustrate useful things to know about Unix. It's common these days to run Unix on an old, spare machine where it's not worth the trouble of upgrading the disks. You may also be dealing with a Unix box at work or school that uses expensive, highly reliable disks with expensive backup procedures in place, where more disk space just isn't an option. It never hurts to know how to eke the last few bytes out of a partition.

This chapter also has a lot of information about compressing and decompressing files, which is fairly common. (These days, you may well compress files to save network bandwidth rather than disk space, but the same principles apply.) So enjoy exploring!

— DH

Sometimes you don't want to remove a file completely — you just want to empty it:

If an active process has the file open (not uncommon for log files), removing the file and creating a new one will not affect the logging program; those messages will just keep going to the file that's no longer linked. Emptying the file doesn't break the association, and so it clears the file without affecting the logging program.
When you remove a file and create a new one with the same name, the new file will have your default permissions and ownership (Section 50.3). It's better to empty the file now, then add new text later; this won't change the permissions and ownership.
Completely empty files (ones that ls -l says have zero characters) don't take any disk space to store (except the few bytes that the directory entry (Section 10.2) uses).
You can use the empty files as "place markers" to remind you that something was there or belongs there. Some Unix logging programs won't write errors to their log files unless the log files already exist. Empty files work fine for that.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many of the techniques in this chapter aren't nearly as applicable as they once were. At the time of this writing, EIDE disks are about a dollar a gigabyte; even fast-wide SCSI isn't that expensive. Often the solution to running low on disk space is just to buy more.

That said, many of these techniques illustrate useful things to know about Unix. It's common these days to run Unix on an old, spare machine where it's not worth the trouble of upgrading the disks. You may also be dealing with a Unix box at work or school that uses expensive, highly reliable disks with expensive backup procedures in place, where more disk space just isn't an option. It never hurts to know how to eke the last few bytes out of a partition.

This chapter also has a lot of information about compressing and decompressing files, which is fairly common. (These days, you may well compress files to save network bandwidth rather than disk space, but the same principles apply.) So enjoy exploring!

— DH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sometimes you don't want to remove a file completely — you just want to empty it:

If an active process has the file open (not uncommon for log files), removing the file and creating a new one will not affect the logging program; those messages will just keep going to the file that's no longer linked. Emptying the file doesn't break the association, and so it clears the file without affecting the logging program.
When you remove a file and create a new one with the same name, the new file will have your default permissions and ownership (Section 50.3). It's better to empty the file now, then add new text later; this won't change the permissions and ownership.
Completely empty files (ones that ls -l says have zero characters) don't take any disk space to store (except the few bytes that the directory entry (Section 10.2) uses).
You can use the empty files as "place markers" to remind you that something was there or belongs there. Some Unix logging programs won't write errors to their log files unless the log files already exist. Empty files work fine for that.
Empty files hold a "timestamp" (just as files with text do) that shows when the file was last modified. I use empty files in some directories to remind me when I've last done something (backups, printouts, etc.). The find -newer (Section 9.8) command can compare other files to a timestamp file.

Well, you get the idea by now.

How can you empty a file? Watch out: when some editors say that a file has "no lines," they may still append a newline character when writing the file. Just one character still takes a block of disk space to store. Here are some better ways to get a properly empty file:

In Bourne-type shells like sh and bash, the most efficient way is to redirect the output of a null command:
```
$ > afile

                  
```
If the file already exists, that command will truncate the file without needing a subprocess.
Copy the Unix empty file, /dev/null (Section 43.12), on top of the file:
```
% cp /dev/null afile

                  
```
Or just cat it there:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some Unix programs — usually background or daemon programs — insist on writing a log file. You might not want the log file itself as much as you want the disk space that the log file takes. Here are a few tips:

Some programs will write to a log file only if the log file exists. If the program isn't running, try removing the log file.
If you remove a log file and the program recreates it, look for command-line options or a configuration-file setup that tells the program not to make the log file.
If you can't get the program to stop writing the log file, try replacing the log file with a symbolic link to /dev/null (Section 43.12):
```
# rm logfile

# ln -s /dev/null logfile

                  
```
The program won't complain, because it will happily write its log file to /dev/null, which discards everything written to it. (Writing to /dev/null is also known as "throwing it in the bit bucket," since all the bits just go away.) Watch out for programs that run at reboot or those that run from the system crontab (Section 25.2) to truncate and replace the log file. These programs might replace the symbolic link with a small regular file that will start growing again.
Does a system mailbox for a user like bin keep getting mail (Section 1.21) that you want to throw away? You may be able to add a .forward file to the account's home directory with this single line:
```
/dev/null
```
Or add an alias in the system mail alias file that does the same thing:
```
bin: /dev/null
```
If your system has a command like newaliases to rebuild the alias database, don't forget to use it after you make the change.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You might have copies of the same file in several directories for the following reasons:

Several different users need to read it (a data file, a program setup file, a telephone list, etc.).
It's a program that more than one person wants to use. For some reason, you don't want to keep one central copy and put its directory in your search path (Section 27.6).
The file has a strange name or it's in a directory you don't usually use. You want a name that's easier to type, but you can't use mv.

Instead of running cp, think about ln. There are lots of advantages to links (Section 10.3). One big advantage of hard links is that they don't use any disk space. The bigger the file, the more space you save with a link. A symbolic link always takes some disk space, so a hard link might be better for ekeing the most space out of your disk. Of course, you have to use a symbolic link if you want to link across filesystems, and symbolic links are much more obvious to other people, so a symlink is less likely to confuse people. Generally the clarity is worth the little bit of extra disk space.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here are techniques to keep you from creating large files (which can happen by accident, such as with runaway programs). Your shell may be able to set process limits. If you're writing a program in C or another language that has access to kernel system calls, you can set these limits yourself. And there's one more trick you can use.

These limits are passed to child processes. So, if your shell sets a limit, all programs you start from that shell will inherit the limit from their parent process.

Many shells have a built-in command that uses system calls to set resource limits. This is usually done from a shell setup file (Section 3.3), but can also be done from the command line at a shell prompt. To set a maximum file size in C-type shells and zsh , use the command limit filesize max-size. In the Korn shell and bash, use ulimit -f max-size. For example, the following csh and ksh commands keep you from creating any files larger than 2 MB:

% limit filesize 2m

$ ulimit -f 2000

Similarly, on many systems, you can use limit and ulimit to restrict the size of core dump files. Core dumps are generally large files, and if you are not actively developing or debugging, they are often not interesting or useful. To set a maximum size for core dumps, execute one of these commands:

% limit coredumpsize 

                  max-size

$ ulimit -c 

                  max-size

To eliminate core dumps entirely, use 0 (zero) for max-size. Because core dumps are essential for effective debugging, any users who actively debug programs should know the commands unlimit coredumpsize (which removes this restriction in csh ) and

ulimit -c

unlimited

for bash and ksh.

File size limits only apply to processes that are invoked from a shell where the limit is set. For instance,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

gzip is a fast and efficient compression program distributed by the GNU project. The basic function of gzip is to take a file filename, compress it, save the compressed version as filename.gz, and remove the original, uncompressed file. The original file is removed only if gzip is successful; it is very difficult to delete a file accidentally in this manner. Of course, being GNU software, gzip has more options than you want to think about, and many aspects of its behavior can be modified using command-line options.

First, let's say that we have a large file named garbage.txt:

rutabaga% ls -l garbage.txt*

-rw-r--r--   1 mdw      hack       312996 Nov 17 21:44 garbage.txt

If we compress this file using gzip, it replaces garbage.txt with the compressed file garbage.txt.gz. We end up with the following:

rutabaga% gzip garbage.txt

rutabaga% ls -l garbage.txt*

-rw-r--r--   1 mdw      hack       103441 Nov 17 21:48 garbage.txt.gz

Note that garbage.txt is removed when gzip completes.

You can give gzip a list of filenames; it compresses each file in the list, storing each with a .gz extension. (Unlike the zip program for Unix and MS-DOS systems, gzip will not, by default, compress several files into a single .gz archive. That's what tar is for; see Section 15.7.)

Go to http://examples.oreilly.com/upt3 for more information on: gzip

How efficiently a file is compressed depends upon its format and contents. For example, many audio and graphics file formats (such as MP3 and JPEG) are already well compressed, and gzip will have little or no effect upon such files. Files that compress well usually include plain-text files and binary files such as executables and libraries. You can get information on a gzip ped file using gzip -l. For example:

rutabaga% gzip -l garbage.txt.gz

compressed  uncompr. ratio uncompressed_name

   103115    312996  67.0% garbage.txt

To get our original file back from the compressed version, we use gunzip, as in:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In the Unix filesystem, files are stored in blocks. Each nonempty file, no matter how small, takes up at least one block. A directory tree full of little files can fill up a lot of partly empty blocks. A big file is more efficient because it fills all (except possibly the last) of its blocks completely.

The tar (Section 39.2) command can read lots of little files and put them into one big file. Later, when you need one of the little files, you can extract it from the tar archive. Seems like a good space-saving idea, doesn't it? But tar, which was really designed for magnetic tape archives, adds "garbage" characters at the end of each file to make it an even size. So, a big tar archive uses about as many blocks as the separate little files do.

Okay, then why am I writing this article? Because the gzip (Section 15.6) utility can solve the problems. It squeezes files down — compressing them to get rid of repeated characters. Compressing a tar archive typically saves 50% or more. The bzip2 (Section 15.6) utility can save even more.

If your compressed archive is corrupted somehow — say, a disk block goes bad — you could lose access to all of the files. That's because neither tar nor compression utilities recover well from missing data blocks. If you're archiving an important directory, be sure you have good backup copies of the archive.

Making a compressed archive of a directory and all of its subdirectories is easy: tar copies the whole tree when you give it the top directory name. Just be sure to save the archive in some directory that won't be copied — so tar won't try to archive its own archive! I usually put the archive in the parent directory. For example, to archive my directory named project, I'd use the following commands. The .tar.gz extension isn't required, but is just a convention; another common convention is .tgz. I've added the gzip - -best option for more compression — but it can be a lot slower, so use it only if you need to squeeze out every last byte.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Two tools, df and du, report how much disk space is free and how much is used by any given directory. For each filesystem, df tells you the capacity, how much space is in use, and how much is free. By default, it lists both local and remote (i.e., NFS (Section 1.21)) filesystems. Under Linux or BSD Unix, the output from df looks like this:

% df

Filesystem  1K-blocks     Used    Avail Capacity  Mounted on

/dev/ad0s1a     99183    37480    53769    41%    /

/dev/ad2s1e   3943876  1873453  1754913    52%    /home

/dev/ad0s1f   3360437  1763460  1328143    57%    /usr

/dev/ad0s1e    508143    16925   450567     4%    /var

procfs              4        4        0   100%    /proc

toy:/usr     17383462 15470733   522053    97%    /toy

   ...

This report shows information about four local filesystems, the local procfs filesystem, and one remote filesystem (from the system toy). Note that a normal filesystem that is 100% full really has 5 to 10% free space — but only the superuser (Section 1.18) can use this reserved space, and that usually isn't a good idea. The reserved space is primarily for recovering from the disk filling up for some reason; the superuser can still successfully copy files and the like to free up space. Special filesystems often don't do this sort of block reservation; procfs and ISO-9660 (CD-ROM and CD-R) filesystems don't care.

df can be invoked in several other ways:

If you already know that you're interested in a particular filesystem, you can use a command such as df /home or df . (. means "the current directory" (Section 1.16)).
If your system uses NFS and you are interested only in local (non-NFS) filesystems, use the command df -t ufs (most BSDs) or df -t ext2fs (most Linuxes). You should always use this command if remote file servers are down. If you have mounted remote disks that are unavailable, df will be extremely slow or hang completely.
If you are interested in inode usage rather than filesystem data capacity, use the command df -i. This produces a similar report showing inode statistics.

If you are using the older System V filesystem (for example, on Solaris), the report from

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's a quick little command that will compress (Section 15.6) files in the current directory and below. It uses find ( Section 9.2) to find the files recursively and pick the files it should compress:

-size Section 9.14, xargs Section 28.17

% find . ! -perm -0100 -size +1 -type f -print | xargs gzip -v

This command finds all files that are the following:

Not executable (! -perm -0100), so we don't compress shell scripts and other program files.
Bigger than one block, since it won't save any disk space to compress a file that takes one disk block or less. But, depending on your filesystem, the -size +1 may not really match files that are one block long. You may need to use -size +2, -size +1024c, or something else.
Regular files (-type f) and not directories, named pipes, etc.

The -v switch to gzip tells you the names of the files and how much they're being compressed. If your system doesn't have xargs, use the following:

% find . ! -perm -0100 -size +1 -type f -exec gzip -v {} \;

Tune the find expressions to do what you want. Here are some ideas — for more, read your system's find manual page:

! -name \*.gz: Skip any file that's already gzipped (filename ends with .gz ).
-links 1: Only compress files that have no other (hard) links.
-user yourname: Only compress files that belong to you.
-atime +60: Only compress files that haven't been accessed (read, edited, etc.) for more than 60 days.

You might want to put this in a job that's run every month or so by at (Section 25.5) or cron (Section 25.2).

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

After you compile and debug a program, there's a part of the executable binary that you can delete to save disk space. The strip command does the job. Note that once you strip a file, you can't use a symbolic debugger like dbx or gdb on it!

Here's an example. I'll compile a C program and list it. Then I'll strip it and list it again. How much space you save depends on several factors, but you'll almost always save something.

-s Section 9.14

% cc -o echoerr echoerr.c

% ls -ls echoerr

  52 -rwxr-xr-x   1 jerry    24706 Nov 18 15:49 echoerr

% strip echoerr

% ls -ls echoerr

  36 -rwxr-xr-x   1 jerry    16656 Nov 18 15:49 echoerr

The GNU strip has a number of options to control what symbols and sections are stripped from the binary file. Check the strip manpage for specific details of the version you have.

If you know that you want a file stripped when you compile it, your compiler probably has a -s option (which is passed to ld after compilation is complete). If you use ld directly — say, in a makefile (Section 11.10) — use the -s option there.

Here's a shell script named stripper that finds all the unstripped executable files in your bin directory (Section 7.4) and strips them. It's a quick way to save space on your account. (The same script, searching the whole filesystem, will save even more space for system administrators — but watch out for unusual filenames):

xargs Section 28.17

#! /bin/sh

skipug="! -perm -4000 ! -perm -2000"  # SKIP SETUID, SETGID FILES

find $HOME/bin -type f \( -perm -0100 $skipug \) -print |

xargs file |

sed -n '/executable .*not stripped/s/: TAB .*//p' |

xargs -rpl strip

The find (Section 9.2) finds all executable files that aren't setuid or setgid and runs file (Section 12.6) to get a description of each. The sed command skips shell scripts and other files that can't be stripped. sed searches for lines from file like the following:

/usr/local/bin/xemacs: TAB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

No matter how much disk space you have, you will eventually run out. One way the system administrator can force users to clean up after themselves is to impose quotas on disk usage. Many Unixes have quota systems available: check your manual pages with a command like apropos quota.

If you're a user, how do quotas affect you? Sooner or later, you may find that you're over your quota. Quotas are maintained on a per-filesystem basis. They may be placed on disk storage (the number of blocks) and on inodes (the number of files). The quota system maintains the concept of hard and soft limits. When you exceed a soft limit, you'll get a warning (WARNING: disk quota exceeded), but you can continue to accumulate more storage. The warning will be repeated whenever you log in. At some point (i.e., after some number of sessions in which the storage stays above the soft limit), the system loses patience and refuses to allocate any more storage. You'll get a message like

OVER

DISK

QUOTA: NO MORE DISK SPACE. At this point, you must delete files until you're again within the soft limit. Users are never allowed to exceed their hard limit. This design allows you to have large temporary files without penalty, provided that they do not occupy too much disk space long-term.

There may also be a quota on the number of files (i.e., inodes) that you can own per filesystem. It works exactly the same way; you'll get a warning when you exceed the soft limit; if you don't delete some files, the system will eventually refuse to create new files.

The quota command shows a user's quota on each filesystem where quotas have been set. With no option, it displays a line for each system where you're over quota. The -v option shows a line for each system where you have a quota. The output can be a bit confusing on systems with the automounter running, since it mounts things dynamically and uses symlinks to make things appear where you expect them, so the filesystem names may not match the directory names you're accustomed to:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

On some Unix systems, the spell command reads one or more files and prints a list of words that may be misspelled. You can redirect the output to a file, use grep (Section 13.1) to locate each of the words, and then use vi or ex to make the edits. It's also possible to hack up a shell and sed script that interactively displays the misspellings and fixes them on command, but realistically, this is too tedious for most users. (The ispell (Section 16.2) program solves many — though not all — of these problems.)

When you run spell on a file, the list of words it produces usually includes a number of legitimate words or terms that the program does not recognize. spell is case sensitive; it's happy with Aaron but complains about aaron. You must cull out the proper nouns and other words spell doesn't know about to arrive at a list of true misspellings. For instance, look at the results on this sample sentence:

$ cat sample

Alcuin uses TranScript to convert ditroff into

PostScript output for the LaserWriter printerr.

$ spell sample

Alcuin

ditroff

printerr

LaserWriter

PostScript

TranScript

Only one word in this list is actually misspelled.

On many Unix systems, you can supply a local dictionary file so that spell recognizes special words and terms specific to your site or application. After you have run spell and looked through the word list, you can create a file containing the words that were not actual misspellings. The spell command will check this list after it has gone through its own dictionary. On certain systems, your word-list file must be sorted (Section 22.1).

If you added the special terms in a file named dict, you could specify that file on the command line using the + option:

$ spell +dict sample

printerr

The output is reduced to the single misspelling.

The spell command will make some errors based on incorrect derivation of spellings from the root words contained in its dictionary. If you understand how spell works (Section 15.4), you may be less surprised by some of these errors.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

On some Unix systems, the spell command reads one or more files and prints a list of words that may be misspelled. You can redirect the output to a file, use grep (Section 13.1) to locate each of the words, and then use vi or ex to make the edits. It's also possible to hack up a shell and sed script that interactively displays the misspellings and fixes them on command, but realistically, this is too tedious for most users. (The ispell (Section 16.2) program solves many — though not all — of these problems.)

When you run spell on a file, the list of words it produces usually includes a number of legitimate words or terms that the program does not recognize. spell is case sensitive; it's happy with Aaron but complains about aaron. You must cull out the proper nouns and other words spell doesn't know about to arrive at a list of true misspellings. For instance, look at the results on this sample sentence:

$ cat sample

Alcuin uses TranScript to convert ditroff into

PostScript output for the LaserWriter printerr.

$ spell sample

Alcuin

ditroff

printerr

LaserWriter

PostScript

TranScript

Only one word in this list is actually misspelled.

On many Unix systems, you can supply a local dictionary file so that spell recognizes special words and terms specific to your site or application. After you have run spell and looked through the word list, you can create a file containing the words that were not actual misspellings. The spell command will check this list after it has gone through its own dictionary. On certain systems, your word-list file must be sorted (Section 22.1).

If you added the special terms in a file named dict, you could specify that file on the command line using the + option:

$ spell +dict sample

printerr

The output is reduced to the single misspelling.

The spell command will make some errors based on incorrect derivation of spellings from the root words contained in its dictionary. If you understand how spell works (Section 15.4), you may be less surprised by some of these errors.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The original Unix spell-checking program, spell ( Section 15.1), is fine for quick checks of spelling in a short document, but it makes you cry out for a real spellchecker, which not only shows you the misspelled words in context, but offers to change them for you.

Go to http://examples.oreilly.com/upt3 for more information on: ispell

ispell, a very useful program that's been ported to Unix and enhanced over the years, does all this and more. Either it will be preinstalled or you'll need to install it for your Unix version.

Here's the basic usage: just as with spell, you spell check a document by giving ispell a filename. But there the similarities cease. ispell takes over your screen or window, printing two lines of context at the bottom of the screen. If your terminal can do reverse video, the offending word is highlighted. Several alternate possibilities are presented in the upper-left corner of the screen — any word in ispell's dictionary that differs by only one letter, has a missing or extra letter, or transposed letters.

Faced with a highlighted word, you have eight choices:

SPACE: Press the spacebar to accept the current spelling.
A: Type A to accept the current spelling, now and for the rest of this input file.
I: Type I to accept the current spelling now and for the rest of this input file and also to instruct ispell to add the word to your private dictionary. By default, the private dictionary is the file .ispell_words in your home directory, but it can be changed with the -p option or by setting the environment variable (Section 35.3) WORDLIST to the name of some other file. If you work with computers, this option will come in handy since we use so much jargon in this business! It makes a lot more sense to "teach" all those words to ispell than to keep being offered them for possible correction. (One gotcha: when specifying an alternate file, you must use an

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Are you writing a document and want to check the spelling of a word before you finish (if you aren't using a word processor with automatic spelling correction, that is)? A Unix system gives you several ways to do this.

Because this is Unix, you can use any of these approaches when you write a script of your own.

If you aren't sure which of two possible spellings is right, you can use the spell command with no arguments to find out. Type the name of the command, followed by a RETURN, then type the alternative spellings you are considering. Press CTRL-d (on a line by itself) to end the list. The spell command will echo back the word(s) in the list that it considers to be in error:
```
$ spell

misspelling

mispelling

CTRL-d

mispelling
```
If you're using ispell ( Section 16.2) or the newer aspell, you need to add the -a option. The purpose of this option is to let the speller interact with other programs; there are details in the programs' documentation. But, like most Unix filters, you can also let these programs read a word from standard input and write their response on standard output; it will either tell you that the spelling is right or give you a list of suggestions. aspell and ispell will use their local dictionaries and improved spelling rules.
As an example, let's check the spelling of outragous and whut with both ispell and aspell:
```
$ ispell -a

@(#) International Ispell Version 3.1.20 10/10/95

outragous whut

& outragous 1 0: outrageous

& whut 5 10: hut, shut, what, whet, whit



CTRL-d

$ aspell -a

@(#) International Ispell Version 3.1.20 (but really Aspell .32.6 alpha)

outragous whut

& outragous 3 0: outrageous, outrages, outrage's

& whut 5 10: what, whet, whit, hut, shut



CTRL-d

$
```
When these spellers start, they print a version message and wait for input. I type the words I want to check and press RETURN. The speller returns one result line for each word:
- A result of * means the word is spelled correctly.
- A line starting with & means the speller has suggestions. Then it repeats the word, the number of suggestions it has for that word, the character position that the word had on the input line, and finally the suggestions.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[If you have ispell (Section 16.2), there's not a whole lot of reason for using spell any more. Not only is ispell more powerful, it's a heck of a lot easier to update its spelling dictionaries. Nonetheless, we decided to include this article, because it clarifies the kinds of rules that spellcheckers go through to expand on the words in their dictionaries. — TOR]

On many Unix systems, the directory /usr/lib/spell contains the main program invoked by the spell command along with auxiliary programs and data files.

On some systems, the spell command is a shell script that pipes its input through deroff -w and sort -u ( Section 22.6) to remove formatting codes and prepare a sorted word list, one word per line. On other systems, it is a standalone program that does these steps internally. Two separate spelling lists are maintained, one for American usage and one for British usage (invoked with the -b option to spell). These lists, hlista and hlistb, cannot be read or updated directly. They are compressed files, compiled from a list of words represented as nine-digit hash codes. (Hash coding is a special technique used to search for information quickly.)

The main program invoked by spell is spellprog. It loads the list of hash codes from either hlista or hlistb into a table, and it looks for the hash code corresponding to each word on the sorted word list. This eliminates all words (or hash codes) actually found in the spelling list. For the remaining words, spellprog tries to derive a recognizable word by performing various operations on the word stem based on suffix and prefix rules. A few of these manipulations follow:

-y+iness +ness -y+i+less +less -y+ies -t+ce -t+cy

The new words created as a result of these manipulations will be checked once more against the spell table. However, before the stem-derivative rules are applied, the remaining words are checked against a table of hash codes built from the file hstop. The stop list contains typical misspellings that stem-derivative operations might allow to pass. For instance, the misspelled word

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

ispell ( Section 16.2) uses two lists for spelling verification: a master word list and a supplemental personal word list.

The master word list for ispell is normally the file /usr/local/lib/ispell/ispell.hash, though the location of the file can vary on your system. This is a "hashed" dictionary file. That is, it has been converted to a condensed, program-readable form using the buildhash program (which comes with ispell) to speed the spell-checking process.

The personal word list is normally a file called .ispell_english or .ispell_words in your home directory. (You can override this default with either the -p command-line option or the WORDLIST environment variable (Section 35.3).) This file is simply a list of words, one per line, so you can readily edit it to add, alter, or remove entries. The personal word list is normally used in addition to the master word list, so if a word usage is permitted by either list it is not flagged by ispell.

Custom personal word lists are particularly useful for checking documents that use jargon or special technical words that are not in the master word list, and for personal needs such as holding the names of your correspondents. You may choose to keep more than one custom word list to meet various special requirements.

You can add to your personal word list any time you use ispell: simply use the I command to tell ispell that the word it offered as a misspelling is actually correct, and should be added to the dictionary. You can also add a list of words from a file using the ispell -a ( Section 16.3) option. The words must be one to a line, but need not be sorted. Each word to be added must be preceded with an asterisk. (Why? Because ispell -a has other functions as well.) So, for example, we could have added a list of Unix utility names to our personal dictionaries all at once, rather than one-by-one as they were encountered during spell checking.

Obviously, though, in an environment where many people are working with the same set of technical terms, it doesn't make sense for each individual to add the same word list to his own private

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The wc (word count) command counts the number of lines, words, and characters in the files you specify. (Like most Unix utilities, wc reads from its standard input if you don't specify a filename.) For example, the file letter has 120 lines, 734 words, and 4,297 characters:

% wc letter

     120     734    4297 letter

You can restrict what is counted by specifying the options -l (count lines only), -w (count words only), and -c (count characters only). For example, you can count the number of lines in a file:

% wc -l letter

     120 letter

or you can count the number of files in a directory:

% cd man_pages

% ls | wc -w

     233

The first example uses a file as input; the second example pipes the output of an ls command to the input of wc. (Be aware that the -a option (Section 8.9) makes ls list dot files. If your ls command is aliased (Section 29.2) to include -a or other options that add words to the normal output — such as the line total nnn from ls -l — then you may not get the results you want.)

The following command will tell you how many more words are in new.file than in old.file:

% expr `wc -w < new.file`  -  `wc -w < old.file`

Many shells have built-in arithmetic commands and don't really need expr ; however, expr works in all shells.

In a programming application, you'll usually want wc to read the input files by using a < character, as shown earlier. If instead you say:

% expr `wc -w new.file` - `wc -w old.file`

the filenames will show up in the expressions and produce a syntax error.

Taking this concept a step further, here's a simple shell script to calculate the differences in word count between two files:

count_1=`wc -w < $1`   # number of words in file 1

count_2=`wc -w < $2`   # number of words in file 2



diff_12=`expr $count_1 - $count_2`   # difference in word count



# if $diff_12 is negative, reverse order and don't show the minus sign:

case "$diff_12" in

-*) echo "$2 has `expr $diff_12 : '-\(.*\)'` more words than $1" ;;

*)  echo "$1 has $diff_12 more words than $2" ;;

esac

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One type of error that's hard to catch when proofreading is a doubled word. It's hard to miss the double "a" in the title of this article, but you might find yourself from time to time with a "the" on the end of one line and the beginning of another.

We've seen awk scripts to catch this, but nothing so simple as this shell function. Here are two versions; the second is for the System V version of tr ( Section 21.11):

uniq Section 21.20

ww( ) { cat $* | tr -cs "a-z'" "\012" | uniq -d; }



ww( ) { cat $* | tr -cs "[a-z]'" "[\012*]" | uniq -d; }

In the script ww.sh , the output of the file is piped to tr to break the stream into separate words, which is then passed to the uniq command for testing of duplicate terms.

—TOR and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A common problem in text processing is making sure that items that need to occur in pairs actually do so.

Most Unix text editors include support for making sure that elements of C syntax such as parentheses and braces are closed properly. Some editors, such as Emacs (Section 19.1) and vim Section 17.1), also support syntax coloring and checking for text documents -- HTML and SGML, for instance. There's much less support in command-line utilities for making sure that textual documents have the proper structure. For example, HTML documents that start a list with <UL> need a closing </UL>.

Unix provides a number of tools that might help you to tackle this problem. Here's a gawk script written by Dale Dougherty that makes sure <UL> and </UL> tags macros come in pairs:

gawk Section 20.11

#! /usr/local/bin/gawk -f

BEGIN {

    IGNORECASE = 1

    inList = 0

    LSlineno = 0

    LElineno = 0

    prevFile = ""

}

# if more than one file, check for unclosed list in first file

FILENAME != prevFile {

    if (inList)

     printf ("%s: found <UL> at line %d without </UL> before end of file\n",

            prevFile, LSlineno)

    inList = 0

    prevFile = FILENAME

}

# match <UL> and see if we are in list

/^<UL>/ {

    if (inList) {

        printf("%s: nested list starts: line %d and %d\n",

            FILENAME, LSlineno, FNR)

    }

    inList = 1

    LSlineno = FNR

}

/^<\/UL>/ {

    if (! inList)

        printf("%s: too many list ends: line %d and %d\n",

            FILENAME, LElineno, FNR)

    else

        inList = 0

    LElineno = FNR

}

# this catches end of input

END {

    if (inList)

        printf ("%s: found <UL> at line %d without </UL> before end of file\n",

            FILENAME, LSlineno)

}

You can adapt this type of script for any place you need to check for a start and finish to an item. Note, though, that not all systems have gawk preinstalled. You'll want to look for an installation of the utility for your system to use this script.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In various textual-analysis scripts, you sometimes need just the words (Section 16.7).

I know two ways to do this. The deroff command was designed to strip out troff Section 45.11) constructs and punctuation from files. The command deroff -w will give you a list of just the words in a document; pipe to sort -u (Section 22.6) if you want only one of each.

deroff has one major failing, though. It considers a word as just a string of characters beginning with a letter of the alphabet. A single character won't do, which leaves out one-letter words like the indefinite article "A."

A substitute is tr (Section 21.11), which can perform various kinds of character-by-character conversions.

To produce a list of all the individual words in a file, type the following:

< Section 43.1

% tr -cs A-Za-z '\012' <

                file

The -c option "complements" the first string passed to tr; -s squeezes out repeated characters. This has the effect of saying: "Take any nonalphabetic characters you find (one or more) and convert them to newlines (\012)."

(Wouldn't it be nice if tr just recognized standard Unix regular expression syntax (Section 32.4)? Then, instead of -c A-Za-z, you'd say '[^A-Za-z]'. It's no less obscure, but at least it's used by other programs, so there's one less thing to learn.)

The System V version of tr ( Section 21.11) has slightly different syntax. You'd get the same effect with this:

% tr -cs '[A-Z][a-z]' '[\012*]' < 

               file

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We're giving a lot of pages to the vi editor. People who use another editor, like Emacs, might wonder why. Here's why.

I've watched people (including myself) learn and use vi for 20 years. It's the standard editor that comes with almost every Unix system these days, but most people have no idea that vi can do so much. People are surprised, over and over, when I show them features that their editor has. Even with its imperfections, vi is a power tool. If you work with files, you probably use it constantly. Knowing how to use it well will save you lots of time and work.

But why not give the same coverage to another editor that lots of people use: GNU Emacs ( Section 19.1)? That's because GNU Emacs comes with source code and can be extended by writing LISP code. Its commands have descriptive names that you can understand by reading through a list. vi's commands are usually no more than a few characters long; many of the option names are short and not too descriptive either. Lots of Unix systems don't even have vi source code these days.

I hope that you vi users will learn a lot in this section and that people who don't use vi will at least browse through to see some of vi's less obvious features.

If you're looking for additional text-editing power, you can use vim instead of the plain vanilla vi installed on most systems. All vi commands work with vim, but with added functionality, power, and more standardized behavior accross flavors of Unix. There should be an installation of vim for your Unix.

—JP and SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We're giving a lot of pages to the vi editor. People who use another editor, like Emacs, might wonder why. Here's why.

I've watched people (including myself) learn and use vi for 20 years. It's the standard editor that comes with almost every Unix system these days, but most people have no idea that vi can do so much. People are surprised, over and over, when I show them features that their editor has. Even with its imperfections, vi is a power tool. If you work with files, you probably use it constantly. Knowing how to use it well will save you lots of time and work.

But why not give the same coverage to another editor that lots of people use: GNU Emacs ( Section 19.1)? That's because GNU Emacs comes with source code and can be extended by writing LISP code. Its commands have descriptive names that you can understand by reading through a list. vi's commands are usually no more than a few characters long; many of the option names are short and not too descriptive either. Lots of Unix systems don't even have vi source code these days.

I hope that you vi users will learn a lot in this section and that people who don't use vi will at least browse through to see some of vi's less obvious features.

If you're looking for additional text-editing power, you can use vim instead of the plain vanilla vi installed on most systems. All vi commands work with vim, but with added functionality, power, and more standardized behavior accross flavors of Unix. There should be an installation of vim for your Unix.

—JP and SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Travel between files, save text into buffers, and move it around without leaving vi: Section 17.3, Section 17.4, and Section 17.6.
Recover deletions from up to nine numbered buffers: Section 17.7.
Do global search and replacement with pattern matching: Section 17.8, Section 17.13, Section 17.14, Section 17.16, and Section 17.22.
Save a lot of typing with word abbreviations: Section 17.23, Section 17.24, and Section 17.25.
"Prettify" lines of text that don't fit on the screen as you would like: Section 17.28.
Run other Unix commands without leaving vi (called a filter-through): Section 17.18 and Section 17.21.
Keep track of functions and included files with ctags and tags.
Change your vi and ex options in your .exrc file for all files or just for files in a local directory: Section 17.5.

When you type a : (colon) command in vi, you're beginning an ex command. There's more information about ex in a later chapter: Section 20.3, Section 20.4, and Section 20.5.

— EK

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

ex commands enable you to switch between multiple files. The advantage is speed. When you are sharing the system with other users, it takes time to exit and re-enter vi for each file you want to edit. Staying in the same editing session and traveling between files is not only faster for access, but you also save abbreviations and command sequences that you have defined, and you keep yank buffers (Section 17.4) so that you can copy text from one file to another.

When you first invoke vi, you can name more than one file to edit and then use ex commands to travel between the files:

% vi file1 file2

This edits file1 first. After you have finished editing the first file, the ex command :w writes (saves) file1, and :n calls in the next file (file2). You can type :wn both to save the current file changes and to go to the next file. Typing :q! discards changes and closes the current file. Type vi * to edit all the files in a directory, though this will give you an error in some Unix systems. Type CTRL-g or :f to get the name of your current file; :args lists all filenames from the command line and puts brackets around the [ current ] file.

You can also switch at any time to another file not specified on the command line with the ex command :e. If you want to edit another file within vi, you first need to save your current file (:w), then you can type the following command:


               :e 

               filename

vi "remembers" two filenames at a time as the current and alternate filenames. These can be referred to by the symbols % (current filename) and # (alternate filename).

# is particularly useful with :e, since it allows you to switch back and forth between two files easily. The command :e# is always "switch to the other one." With different flavors of Unix, the vi command CTRL-^ (control-caret) is a synonym for :e#. This usually seems to work even without pressing the SHIFT key. For instance, if I get a caret by pressing SHIFT-6, I don't need to press CTRL-SHIFT-6 to make

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you give a yank buffer (temporary holding buffer) a one-letter name, you have a convenient way to move text from one file to another. Named buffers are not cleared when a new file is loaded into the vi buffer with the :e command (Section 17.3). Thus, by yanking (copying) or deleting text from one file (into multiple named buffers if necessary), calling in a new file with :e and putting the named buffer into the new file, you can transfer material between files.

The following table illustrates how to transfer text from one file to another. Type the keystrokes exactly as shown to achieve the stated result.

Keystrokes	Action	Results
`"f4yy`	Yank four lines into buffer f.	With a screen editor you can scroll the page, move the cursor, delete lines, insert characters, and more, while seeing the results of the edits as you make them
`:w`	Save the file.	"practice" 6 lines 238 characters
`:e letter`	Enter the file letter with `:e`. Move cursor to where the copied text will be placed.	Dear Mr. Henshaw: I thought that you would be interested to know that: Yours truly,
`"fp`	Place yanked text from named buffer f below the cursor.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In addition to reading the .exrc file (the vi configuration or startup file) in your home directory, many versions of vi will read a file called .exrc in the current directory. This allows you to set options that are appropriate to a particular project.

For example, you might want to have one set of options in a directory used mainly for programming:

set number lisp autoindent sw=4 terse

set tags=/usr/lib/tags

and another set of options in a directory used for text editing:

set wrapmargin=15 ignorecase

Note that you can set certain options in the .exrc file in your home directory (Section 1.15) and unset them (for example, set wrapmargin=0 noignorecase) in a local directory.

Many versions of vi don't read .exrc files in the current directory unless you first set the exrc option in your home directory's .exrc file:

set exrc

This mechanism makes it harder for other people to place, in your working directory, an .exrc file whose commands might jeopardize the security of your system.

You can also define alternate vi environments by saving option settings in a file other than .exrc and reading in that file with the :so command. For example:

:so .progoptions

Local .exrc files are also useful for defining abbreviations ( Section 17.23) and key mappings (Section 18.2). When we write a book or manual, we save all abbreviations to be used in that book in an .exrc file in the directory in which the book is being created.

You can also store settings and startup commands for vi and ex in an environment variable called EXINIT (Section 17.27). If there is a conflict between settings in EXINIT and an .exrc file, EXINIT settings take precedence.

You can keep a group of standard .exrc files in a central directory and link ( Section 10.5) to them from various local directories. For instance, from this book's source-file directory, which is full of SGML files, I made a symlink:

% ln -s ~/lib/vi/exrc.sgml .exrc

I prefer symbolic links to hard links in a case like this because they make it easy to see to which central file the local

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In a vi editing session, your last deletion (d or x) or yank (y) is saved in a buffer. You can access the contents of that buffer and put the saved text back in your file with the put command (p or P). This is a frequent sequence of commands:

5dd   delete 5 lines

                . . . move somewhere else



p     put the 5 deleted lines back in a new

               location, below the current line

Fewer new users are aware that vi stores the last nine (Section 17.7) deletions in numbered buffers. You can access any of these numbered buffers to restore any (or all) of the last nine deletions. (Small deletions, of only parts of lines, are not saved in numbered buffers, however.) Small deletions can be recovered only by using the p or P command immediately after you've made the deletion.

vi also allows you to yank (copy) text to "named" buffers identified by letters. You can fill up to 26 (a-z) buffers with yanked text and restore that text with a put command at any time in your editing session. This is especially important if you want to transfer data between two files, because all buffers except those that are named are lost when you change files. See Section 17.4.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Being able to delete large blocks of text in a single bound is all very well and good, but what if you mistakenly delete 53 lines that you need? There's a way to recover any of your past nine deletions, because they're saved in numbered buffers. The last delete is saved in buffer 1, the second-to-last in buffer 2, and so on.

To recover a deletion, type <"> (the double quote character), identify the buffered text by number, then give the put command. To recover your second-to-last deletion from buffer 2, type the following:

"2p

The deletion in buffer 2 is placed on the line below the cursor.

If you're not sure which buffer contains the deletion you want to restore, you don't have to keep typing <"> n p over and over again. If you use the repeat command (.) with p after u (undo), it automatically increments the buffer number. As a result, you can search through the numbered buffers as follows:

"1pu.u.u etc.

to put the contents of each succeeding buffer in the file one after the other. Each time you type u, the restored text is removed; when you type a dot (.), the contents of the next buffer is restored to your file. Keep typing u and . until you've recovered the text you're looking for.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Besides using line numbers and address symbols (., $, %), ex (including the ex mode of vi, of course) can address lines (Section 20.3) using search patterns ( Section 32.1). For example:

:/ pattern /d: Deletes the next line containing pattern.
:/ pattern /+d: Deletes the line below the next line containing pattern. (You could also use +1 instead of + alone.)
:/ pattern1 /,/ pattern2 /d: Deletes from the next line (after the current line) that contains pattern1 through the next following line that contains pattern2.
:.,/ pattern /m23: Takes text from current line (.) through the next line containing pattern and puts it after line 23.

Note that patterns are delimited by a slash both before and after.

If you make deletions by pattern with vi and ex, there is a difference in the way the two editors operate. Suppose you have in your file named practice the following lines:

With a screen editor you can scroll the

page, move the cursor, delete lines, insert

characters and more, while seeing results

of your edits as you make them.

Key-strokes	Action	Results
`d/while`	The vi delete-to-pattern command deletes from the cursor up to the word while but leaves the remainder of both lines.	With a screen editor you can scroll the page, move the cursor, while seeing results of your edits as you make them.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It makes sense to be overly careful when using a search-and-replace command. It sometimes happens that what you get is not what you expected. You can undo any search-and-replace command by entering u, provided that the command was intended for the most recent edit you made. But you don't always catch undesired changes until it is too late to undo them. Another way to protect your edited file is to save the file with :w before performing a global replacement. Then at least you can quit the file without saving your edits and go back to where you were before the change was made. You can also read back in the previous version of the buffer with :e! (Section 17.3).

It's wise to be cautious and know exactly what is going to be changed in your file. If you'd like to see what the search turns up and confirm each replacement before it is made, add the c option (for confirm) at the end of the substitute command:


               :1,30s/his/the/gc

The item to be substituted is highlighted so that placement of the cursor on the first character is marked by a series of carets (^^^^).

copyists at his school

            ^^^_

If you want to make the replacement, you must enter y (for yes) and press RETURN. If you don't want to make a change, simply press RETURN.

The combination of the vi commands, n (repeat last search) and dot (.) (repeat last command), is also an extraordinarily useful and quick way to page through a file and make repetitive changes that you may not want to make globally. So, for example, if your editor has told you that you're using which when you should be using that, you can spot-check every occurrence of which, changing only those that are incorrect.

This often turns out to be faster than using a global substitution with confirmation. It also lets you see other lines near the text you're checking, which is hard to do with :s///c in original vi. vi clones have improved the situation. For instance, in vim, :s///c runs in fullscreen mode; it also lets you type CTRL-y and CTRL-e to scroll the screen up or down to see context before you approve or deny each substitution.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can use :w to save an entire buffer (the copy of the file you are editing) under a new filename.

Suppose you have a file practice, containing 600 lines. You open the file and make extensive edits. You want to quit but save both the old version of practice and your new edits for comparison. To save the edited buffer in a file called check_me, give the command:


               :w check_me

Your old version, in the file practice, remains unchanged (provided that you didn't previously use :w). You can now quit the old version by typing :q.

— LL, from Learning the vi Editor (O'Reilly, 1998)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

While editing, you will sometimes want to save just part of your file as a separate, new file. For example, you might have entered formatting codes and text that you want to use as a header for several files.

You can combine ex line addressing (Section 20.3) with the write command, w, to save part of a file. For example, if you are in the file practice and want to save part of practice as the file newfile, you could enter:

:230,$w newfile: Saves from line 230 to end-of-file in newfile.
:.,600w newfile: Saves from the current line to line 600 in newfile.

After newfile has been created, you'll need w! instead of w.

— LL, from Learning the vi Editor (O'Reilly, 1998)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can use the Unix redirect and append operator (>>) with w to append all or part of the buffer's contents to an existing file. For example, if you entered:


               :1,10w 

               newfile

and then:

$ Section 20.3


               :340,$w >>

               newfile

newfile would contain lines 1-10 and line 340 to the end of the buffer.

—TOR, from Learning the vi Editor (O'Reilly, 1998)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can move blocks of text delimited by patterns (Section 17.8). For example, assume you have a 150-page reference manual. All reference pages are organized into three paragraphs with the same three headings: SYNTAX, DESCRIPTION, and PARAMETERS. A sample of one reference page follows:

 .Rh 0 "Get status of named file" "STAT"

 .Rh "SYNTAX"

 .nf

 integer*4 stat, retval

 integer*4 status(11)

 character*123 filename

 ...

 retval = stat (filename, status)

 .fi

 .Rh "DESCRIPTION"

 Writes the fields of a system data structure into the

 status array.

 These fields contain (among other

 things) information about the file's location, access

 privileges, owner, and time of last modification.

 .Rh "PARAMETERS"

 .IP "\fBfilename\fR" 15n

 A character string variable or constant containing

 the Unix pathname for the file whose status you want

 to retrieve.

 You can give the ...

Suppose that it is decided to move the SYNTAX paragraph below the DESCRIPTION paragraph. Using pattern matching, you can move blocks of text on all 150 pages with one command!


               :g/SYNTAX/,/DESCRIPTION/-1 mo /PARAMETERS/-1

This command operates on the block of text between the line containing the word SYNTAX and the line just before the word DESCRIPTION (/DESCRIPTION/-1). The block is moved (using mo) to the line just before PARAMETERS (/PARAMETERS/-1). Note that ex can only place text below the line specified. To tell ex to place text above a line, you first have to move up a line with -1 and then place your text below. In a case like this, one command literally saves hours of work. (This is a real-life example — we once used a pattern match like this to rearrange a reference manual containing hundreds of pages.)

Block definition by patterns can be used equally well with other ex commands. For example, if you wanted to delete all DESCRIPTION paragraphs in the reference chapter, you could enter:


               :g/DESCRIPTION/,/PARAMETERS/-1d

This very powerful kind of change is implicit in

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The best way to learn pattern matching is by example, so here's a short list of pattern-matching examples with explanations. (Section 32.21 has a list of these patterns.) Study the syntax carefully so you understand the principles at work. You should then be able to adapt these examples to your own situation.

Change all occurrences of the word help (or Help) to HELP:
% Section 20.3
```
                     :%s/[Hh]elp/HELP/g

                  
```
or:
```
                     :%s/[Hh]elp/\U&/g

                  
```
The \U changes the pattern that follows to all uppercase. The pattern that follows is the repeated search pattern, which is either help or Help.
Replace one or more spaces following a colon (:) or a period (.) with two spaces (here a space is marked by a ·):
```
                     :%s/$[:.]$··*/\1··/g
```
Either of the two characters within brackets can be matched. This character is saved into a hold buffer, using $ and $ (Section 34.11) and restored on the right-hand side by the \1. Note that most metacharacters lose their special meanings inside brackets — so the dot does not need to be escaped with a backslash (\).
Delete all blank lines:
g Section 20.4
```
                     :g/^$/d

                  
```
What you are actually matching here is the beginning of the line (^), followed by the end of the line ($), with nothing in between.
Delete all blank lines, plus any lines that contain only whitespace:
```
                     :g/^[·tab

                     ]*$/d

                  
```
(In the previous line, a TAB character is shown as tab.) A line may appear to be blank, but may in fact contain spaces or tabs. The previous numbered example will not delete such a line. This example, like the previous one, searches for the beginning and end of the line. But instead of having nothing in between, the pattern tries to find any number of spaces or tabs. If no spaces or tabs are matched, the line is blank. To delete lines that contain whitespace but that

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Want to see how many times you used the word very in a file? There are a couple of easy ways.

First, tell vi to stop searching when you get to the end of the file. Type the command

:set

nowrapscan

or put it in your .exrc file (Section 17.30).

Move to the top of the file with the 1G command. Search for the first very with the command /very (HINT: using the word-limiting regular expression /\<very\> (Section 32.12) instead will keep you from matching words like every). To find the next very, type the n (next) command.
When vi says Address search hit BOTTOM without matching pattern, you've found all of the words.

Use the command:


                     :g/very/p

The matching lines will scroll down your screen.

To find the line numbers, too, type :set number before your searches.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Are you typing the title of an article or something else that needs an uppercase letter at the start of every word? Do you need to capitalize some text that isn't? It can be tedious to press the SHIFT key as you enter the text or to use ~ (tilde) and w commands to change the text. The following command capitalizes the first character of every word.

:s/\<./\u&/g

(You might be wondering why we didn't use :s/\<[a-z]/\u&/g to match lowercase letters. The <. actually matches the first character of every word, but the \u will only affect letters. So, unless you only want to capitalize certain letters, <. is enough.)

The previous example does only the current line. You can add a range of lines after the colon. For example, to edit all lines in the file, type the following:


               :%s/\<./\u&/g

To do the current line and the next five, use this:


               :.,+5s/\<./\u&/g

To make the first character of each word uppercase (with \u) and the rest lowercase (with \L), try:

$...$...\1 Section 32.21


               :s/\<\(.\)\([A-Za-z]*\)\>/\u\1\L\2/g

The previous command doesn't convert the back ends of words with hyphens (like CD-ROM) or apostrophes (like O'Reilly) to lowercase. That's because [A-Za-z]*\> only matches words whose second through last characters are all letters. You can add a hyphen or an apostrophe to make that expression match more words, if you'd like.

Those commands can be a pain to type. If you use one of them a lot, try putting it in a keymap (Section 18.2).

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Do you need to set certain editor options for certain files — but not use the same setup for every file you edit? Make a special setup file with the same name and an underscore ( _ ) or an extension like .vi, .ex, or .so at the end. For instance, a file named report could have a corresponding setup file named report_ or report.so. (You don't have to use an underscore at the end of the filename. It's convenient, though, because it's not a shell special character (Section 27.17).)

The setup file has the same format as a .exrc file (Section 17.5). To make the editor read it, map (Section 18.2) a function key like F1 (or any other key sequence):

source Section 20.4, ^[ Section 18.6

map #1 :source %_^[

When you start vi, tap that key to read the setup file. (The percent sign stands for the current filename (Section 17.3).)

If you want to use the same setup file for several files in a directory, you might want to make hard links (Section 10.4) between them. That will save disk space. It also means that if you decide to change a setup option, you can edit one of the links to the setup file, and the others will have the same change.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you're editing in vi, you can send a block of text as standard input to a Unix command. The output from this command replaces the block of text in the buffer.

In vi, you can filter text through a Unix command by typing an exclamation mark (! ) followed by any of vi's movement keystrokes that indicate a block of text and then by the Unix command line to be executed. For example:


               !)

               command

will pass the next sentence through command.

There are a couple of unusual features about how vi acts when you use this structure:

First, the exclamation mark doesn't appear on your screen right away. When you type the keystroke(s) for the text object you want to filter, the exclamation mark appears at the bottom of the screen, but the character you type to reference the object does not.
Second, text blocks must be more than one line, so you can use only the keystrokes that would move more than one line ( G, { }, ( ), [[ ]], +, - ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both !10+ and 10!+ would indicate the next ten lines.) Objects such as w do not work unless enough of them are specified so as to exceed a single line. You can also use a slash (/) followed by a pattern and a carriage return to specify the object. This takes the text up to the pattern as input to the command.
Third, there is a special text object that can be used only with this command syntax; you can specify the current line by entering a second exclamation mark:
```
                     !!

                     command

                  
```
Remember that either the entire sequence or the text object can be preceded by a number to repeat the effect. For instance, to change lines 96 through 99 as in the previous example, you could position the cursor on line 96 and enter either:
```
                     4!!sort

                  
```
or:
```
                     !4!sort

                  
```

As another example, assume you have a portion of text in a message that you'd like to convert to all uppercase letters.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Have you ever used the vi -r command to recover a file? It lets you get a file back that you were editing when the system crashed or something else killed your editor before you could save. The system may send you an email message something like this:

Date: Thu, 19 Nov 1999 09:59:00 EST

To: jerry

A copy of an editor buffer of your file "afile"

was saved when the system went down.

This buffer can be retrieved using the "recover" command of the editor.

An easy way to do this is to give the command "vi -r afile".

This works for "edit" and "ex" also.

Section 17.20

Your files are saved under a directory named something like /usr/preserve. Follow the instructions and you'll get back your file, more or less the way it was when you lost it.

If your computers have networked filesystems, such as NFS, there's a wrinkle in the way that vi -r works. It may only work right on the specific computer where you were editing a file. For example, if you're editing the file foo on the host named artemis and it crashes, you may not be able to log on to another host and do vi -r foo to recover that file. That's because, on many hosts, temporary files (like editor buffers) are stored on a local filesystem instead of on the networked (shared) filesystems. On this kind of system, you may need to log on to artemis to recover your lost editor buffer.

If you don't remember which computer you were using when the file was lost, check the "Received:" lines in the email message header; they'll often show from which machine the message originally came. Also, if you don't remember what files are saved on a machine, you can usually get a list of your saved files by typing vi -r without a filename:

% vi -r

/var/preserve/jerry:

On Wed Jul 17 at 08:02 saved 15 lines of file "/u/jerry/Mail/drafts/1"

On Sun Aug 25 at 18:42 saved 157 lines of file "doit"

/tmp:

No files saved.

Don't wait too long. Many Unix systems remove these saved editor buffers every month, week, or sooner.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Usually, when you're editing a file with vi, if you type the command ZZ, it saves your file. But if you recover a file with vi -r (Section 17.19), typing ZZ may not save your edits!

That might be a good thing. When you recover a buffer, you need to decide whether the recovered buffer is really what you want. Maybe you've made other changes to the file since then. Maybe something went wrong as the buffer was being saved (say, the system crashed). You shouldn't just save without checking first.

You can use the :w! command to write the recovered version after you're sure that you want it. Use the :q! command if you don't want the recovered version.

Another good choice is to write the recovered buffer using a different filename, then compare the recovered buffer to the original file. For example, here I recover a draft MH email message and immediately write it to a file named recovered-9 in my tmp directory. Then I use a shell escape (Section 17.21) to run diff (Section 11.1) and compare the draft file on disk (/home/jerry/Mail/drafts/9) with the copy of the recovered buffer that I just wrote (/home/jerry/tmp/recovered-9); the vi current filename % and alternate filename # shortcuts (Section 17.3) are handy here. Oops: diff shows that the recovered version has replaced the last three lines of the message on disk, in the recovered version, with more than 2,000 lines of junk!

less Section 12.3

% vi -r /home/jerry/Mail/drafts/9

               ...recovered file appears...

               :w ~/tmp/recovered-9

/home/jerry/tmp/recovered-9: 55 lines, 168767 characters.

:!diff % # | less

!diff /home/jerry/Mail/drafts/9 /home/jerry/tmp/recovered-9 | less

5c5

< Subject: Re: Two more Unix Power Tools questions

---

> Subject: Next UPT (was: Re: Two more Unix Power Tools questions)

146,148c146,2182

< Yes, you mentioned it once.  Thanks for pointing that out, Greg.

< I think the next job is to review all the articles in that chapter

< to be sure which items should be included -- just the articles, or

---

> Yes, you^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some Unix commands (usually interactive commands like vi) let you run another Unix command temporarily. To do that, you type a special command character — usually an exclamation point (!) — then type the Unix command line you want to run. In this article, I'll show examples for the vi editor. To see if this works on another utility, check its documentation or just try typing ! Unixcommand when the utility is waiting for you to type a command.

You can run any Unix command without quitting vi. That's handy, for example, if you want to read your mail or look at some other file . . . , then go back to the file you were editing without losing your place. It's called a "shell escape." (By the way, there's a another way to do this, job control ( Section 23.3), that works on most Unix systems. Job control is often more convenient and flexible than shell escapes.)

Let's say you're editing the file named foo and you need to run grep to get someone's phone number from your phone file. The steps are as follows:

Be sure you're in command mode (press the ESC key if you aren't sure).
If you want to run a command that needs the file you're editing, remember to write out your vi buffer with the :w command. (So you probably wouldn't need to write anything before the following grep command.) Type :! followed by the Unix command, then press RETURN. For example:
```
                     :!grep tim ~/phone

                  
```
The grep program will run. When it finishes, vi will say:
```
[Hit return to continue]
```
After you press RETURN, you'll be right back where you were.

Other examples:

:!less afile: Page through afile on your screen.
:!rcsdiff %: Give this file to the rcsdiff (Section 11.3) program to see what you've changed since the file was checked out of the archive. vi replaces % with the name of the file you're editing now (Section 17.3).
:!mail

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You probably know that you can search for a word or phrase with the vi / (slash) command:


               /treasure

If you have a file that uses the same word over and over again, you might want to find one particular place that the word is used. You can repeat the search with the n command until you find the place you want. That can take time and effort, though.

For example, suppose you want to find the word "treasure" in the sentence that has words like "Los Alamos residents . . . treasure," but you can't remember exactly how the sentence is written. You could use wildcards in your regular expression:


               /Los Alamos.*treasure

but then the phrases "Los Alamos" and "treasure" have to be on the same line of the file you're searching — and they won't always be. Also, you want your cursor on the word treasure, but that search would put the cursor on Los instead.

"Hmmm," you say, "How about two separate searches, like this?"

/Los Alamos

/treasure

The problem there is that the file might have the phrase "Los Alamos" all throughout it; you might have to type n over and over until you get to the sentence with treasure.

Here's the easy way: a compound search. Say your cursor is on line 1 of the following file:

Before the second World War, there was a treasured boys' school in

what was to become the city of Los Alamos, New Mexico. The school at

Los Alamos changed the lives and made a lifelong impression on most boys

who attended. One of the boys who attended the Los Alamos school went on

to propose that remote set of mesas as a site for the U.S. Government's

   ...

Since the war ended, most of the boys' school ranch buildings have been torn

down or replaced. But there's one building that Los Alamos residents still

use and treasure. It's The Lodge, a log building on the edge of what's now

   ...

Type the command:


               /Los Alamos/;/treasure/

That means "find the first occurrence of treasure just after Los Alamos." Starting at the top of the previous example, that search will skip past all the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can define abbreviations that vi will automatically expand into the full text whenever it's typed during text-input mode. To define an abbreviation, use the ex command:


               :ab 

               abbr phrase

abbr is an abbreviation for the specified phrase. The sequence of characters that make up the abbreviation will be expanded during text-input mode only if you type it as a full word; abbr will not be expanded within a word. [I abbreviate Covnex to Convex, my company's name, because I have dyslexic fingers. -- TC]

Suppose you want to enter text that contains a frequently occuring phrase, such as a difficult product or company name. The command:


               :ab ns the Nutshell Handbook

abbreviates the Nutshell Handbook to the initials ns. Now whenever you type ns as a separate word during text-input mode, ns expands to the full text.

Abbreviations expand as soon as you press a nonalphanumeric character (e.g., punctuation), a carriage return, or ESC (returning to command mode). When you are choosing abbreviations, choose combinations of characters that don't ordinarily occur while you are typing text. If you create an abbreviation that ends up expanding in places where you don't want it to, you can disable the abbreviation by typing:


               :unab 

               abbr

To list your currently defined abbreviations, type:

:ab

The characters that compose your abbreviation cannot appear at the end of your phrase. For example, if you issue the command:


               :ab PG This movie is rated PG

you'll get the message No tail recursion, and the abbreviation won't be set. The message means that you have tried to define something that will expand itself repeatedly, creating an infinite loop. If you issue the command:


               :ab PG the PG rating system

you may or may not produce an infinite loop, but in either case you won't get a warning message. For example, when the previous command was tested on a System V version of Unix, the expansion worked. On a Berkeley version, though, the abbreviation expanded repeatedly, like this:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The vi command ab (Section 17.23) is for abbreviating words. But it's also good for abbreviating ex-mode commands that you type over and over. In fact, for ex-mode commands (commands that start with a colon (:)), abbreviations can be better than keymaps (Section 18.2). That's because you can choose almost any command name; you don't have to worry about conflicts with existing vi commands.

Here's an example. If you have a windowing terminal or more than one terminal, you might have vi sessions running in more than one place. Your system might have a way to transfer text between windows, but it can be easier to use files in /tmp — especially for handling lots of text. (If your text is confidential and your umask (Section 49.4) isn't set to make new files unreadable by other users, try using a more private directory.) Here are some abbreviations from my .exrc (Section 17.30) file:

Go to http://examples.oreilly.com/upt3 for more information on: exrc

ab aW w! /tmp/jerry.temp.a

ab aR r /tmp/jerry.temp.a

ab bW w! /tmp/jerry.temp.b

ab bR r /tmp/jerry.temp.b

   ...

I use those abbreviations this way. To write the current and next 45 lines to temporary file a, I type this command in one vi session:


               :.,+45 aW

To read those saved lines into another vi session, I use:

:aR

You can do the same thing in a single vi session by using named buffers (Section 17.4), but temporary files are the only method that works between two separate vi sessions.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Abbreviations (Section 17.23) are a handy way to fix common typos. Try a few abbreviations like this:

ab teh the

ab taht that

in your .exrc (Section 17.5) file.

Any time you find yourself transposing letters or saying, "Darn, I always misspell that word," add an abbreviation to .exrc. (Of course, you do have to worry about performance if the file gets too big.)

You may be able to enforce conventions this way. For example, command names should be surrounded by <command> tags, so creating a list of abbreviations like this:

ab vi <command>vi</command>

saves us from having to type lots of SGML codes.

(Depending on your version of vi, this abbreviation may be recursive (Section 17.23) because the vi is sandwiched between other nonalphanumeric characters. nvi repeated the <command>) quite a few times and quit, but vim did what we wanted.)

—TOR and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[Quite a few vi users understand how to build vi commands with the (number)(command)(text object) model. But not too many people understand the difference between line commands and character commands. This article explains that and gives some examples. — JP]

The _ (underscore) command is very similar to the ^ (caret) command in that it moves to the first nonblank character of the current line. The key difference is that _ is a line command while ^ is a character command. This is important for all functions that read an "address" — for example, d, y, and c.

In fact, delete, yank, and so on all call a common internal routine in vi to get an "address." If the address is of a particular character, vi does a character-mode delete or yank or whatever. If it is a line address, vi does a line-mode operation. The "address" command may be any of the regular positioning commands (e.g., W, b, $, or /pattern/) or the original character repeated (as in dd or yy).

Some examples are found in Table 17-1.

Table 17-1: Examples of vi character and line commands
Keystrokes	Results
`dd`	Deletes the current line.
`d'a`	Deletes all lines between the current line and the line containing mark `a`, inclusive.
`d'a`	Deletes all characters between the current character and the character at mark `a`. This works much like an Emacs `W` in that the two endpoints are considered to be between two characters. Note that a character-oriented delete may delete newlines.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

vi keeps its temporary copy of the file you're editing in a temporary-file directory — usually /tmp, /usr/tmp, or /var/tmp. If you're editing a big file or if the temporary filesystem runs out of space, vi may not be able to make your temporary file. When that happens, you can use vi's set directory command to set the pathname of a different temporary directory. (If this happens a lot though, you should talk to the system administrator and see if the standard area can be cleaned up or made bigger.)

First, you'll need the absolute pathname (Section 3.7) of a directory on a filesystem with enough room. Use an existing directory, or make a new one.

The vi command is

set

directory

. For example:

set directory=/usr1/jim/vitemp

You have to type that command before giving vi a filename to edit — after that, vi has made the temporary file, and you'll be too late. But if you type that command while using vi and then use the :e command (Section 17.3), all files from then on will use the new temporary directory (in the versions I tried, at least).

To set the directory temporarily, it's probably easiest to add that command to the EXINIT environment variable:

setenv EXINIT 'set directory=/usr1/jim/vitemp'

If you already have a .exrc file (Section 17.5), setting EXINIT will make vi ignore your .exrc file. To make the temporary set directory work, too, use a command with a vertical bar (|), like this:

setenv EXINIT 'source /usr1/jim/.exrc|set directory=/usr1/jim/vitemp'

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Have you made edits that left some of your lines too short or long? The fmt ( Section 21.2) utility can clean that up. Here's an example. Let's say you're editing a file (email message, whatever) in vi and the lines aren't even. They look like this:

This file is a mess

with some short lines

and some lines that are too long — like this one, which goes on and on for quite 

a while and etc.



Let's see what 'fmt' does with it.

You put your cursor on the first line and type (in command mode):

5!! Section 17.18


               5!!fmt

which means "filter (Section 17.18) 5 lines through fmt." Then the lines will look like this:

This file is a mess with some short lines and some lines that are too

long — like this one, which goes on and on for quite a while and etc.



Let's see what 'fmt' does with it.

This is handiest for formatting paragraphs. Put your cursor on the first line of the paragraph and type (in command mode):


               !}fmt

If you don't have any text in your file that needs to be kept as is, you can neaten the whole file at once by typing:

% Section 20.3


               :%!fmt

There are a few different versions of fmt, some fancier than others. Most of the articles in Chapter 21 about editing-related tools can be handy too. For example, recomment reformats program comment blocks. cut (Section 21.14) can remove columns, fields, or shorten lines; tr (Section 21.11) can do other transformations. To neaten columns, try filtering through with the setup in Section 21.17. In general, if the utility will read its standard input and write converted text to its standard output, you can use the utility as a vi filter.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Often, you're editing one part of a file and need to go to another point to look at something. How do you get back?

You can mark your place with the m command. In command mode, type m followed by any letter. (We'll use x in the example.) Here are the commands to do the job:

m x: Marks current position with x (x can be any letter).
' x: Moves cursor to first character of line marked by x.
` x: Moves cursor to character marked by x.
``: Returns to exact position of previous mark or context after a move.
'': Returns to the beginning of the line of the previous mark or context.

I often find it just as easy to type u to undo my last edit. That pops me right back to the place where I was editing. Then I type u again to restore the edit. Watch out for the new multilevel undo feature in vi clones: typing u twice will undo two edits! (I still use m if I want to mark more than one place.)

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can store commands and settings to be executed any time you start the vi or ex editors (Section 17.2) in .exrc in your home directory. You can modify the .exrc file with the vi editor, just as you can any other text file.

If you don't yet have an .exrc file, simply use vi to create one. Enter into this file the set, ab (Section 17.23), and map (Section 18.2) commands that you want to have in effect whenever you use vi or ex. A sample .exrc file looks like this:

set nowrapscan wrapmargin=7

set sections=SeAhBhChDh nomesg

map q :w^M:n^M

" To swap two words, put cursor at start of first word and type v:

map v dwElp

ab ORA O'Reilly & Associates, Inc.

The ^M characters are RETURNs. Make them by pressing CTRL-v, then RETURN (Section 18.6). Lines that start with a double quote (") are comments. Since the file is actually read by ex before it enters vi, commands in .exrc should not have a preceding colon (:).

In addition to reading the .exrc file in your home directory, vi will read the .exrc file in the current directory. This allows you to set options that are appropriate to a particular project (Section 17.5).

If your .exrc file doesn't seem to be working, watch carefully for error messages just as vi starts, before it clears your screen. If you can't read them quickly enough, start ex instead of vi. The q! command quits ex:

% ex

No tail recursion

:q!

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Keymapping — storing complex command sequences so that they can be executed with a single keystroke — is one of my favorite timesavers. There's nothing like typing one key and watching a whole string of work take place. For repetitive edits (e.g., font changes) it's a real wrist-saver, too. In this chapter we show you how to:

Save time by mapping keys: Section 18.2, Section 18.4, Section 18.7, and Section 18.8.
Know when to map a key and when not to: Section 18.3.
Map keys like ESC and RETURN: Section 18.6.
Move around the file without leaving text-input mode: Section 18.11
Protect the text you're pasting in from another window: Section 18.5.
Put custom commands in your .exrc file: Section 18.9 and Section 18.12.
Break long lines of text: Section 18.13.

— EK

While you're editing, you may find that you are using a command sequence frequently, or you may occasionally use a very complex command sequence. To save yourself keystrokes — or the time it takes to remember the sequence — assign the sequence to an unused key by using the map and map! commands.

The map command acts a lot like ab (Section 17.23) except that you define a macro for command mode instead of text-input mode. The map! command works during text-input mode; see the following list.

map x sequence: Define x as a sequence of editing commands.
unmap x: Disable the x definition.
map: List the characters that are currently mapped.

As with other ex-mode commands, these map commands can be saved in your .exrc file (Section 17.30) or typed in after a colon (:). If you want a keymap to use just during this editing session, you might find that vi @-functions (Section 18.4) are easier to create and use. The map commands are best for keymaps that you save in your .exrc file and use during many editing sessions.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Keymapping — storing complex command sequences so that they can be executed with a single keystroke — is one of my favorite timesavers. There's nothing like typing one key and watching a whole string of work take place. For repetitive edits (e.g., font changes) it's a real wrist-saver, too. In this chapter we show you how to:

Save time by mapping keys: Section 18.2, Section 18.4, Section 18.7, and Section 18.8.
Know when to map a key and when not to: Section 18.3.
Map keys like ESC and RETURN: Section 18.6.
Move around the file without leaving text-input mode: Section 18.11
Protect the text you're pasting in from another window: Section 18.5.
Put custom commands in your .exrc file: Section 18.9 and Section 18.12.
Break long lines of text: Section 18.13.

— EK

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

While you're editing, you may find that you are using a command sequence frequently, or you may occasionally use a very complex command sequence. To save yourself keystrokes — or the time it takes to remember the sequence — assign the sequence to an unused key by using the map and map! commands.

The map command acts a lot like ab (Section 17.23) except that you define a macro for command mode instead of text-input mode. The map! command works during text-input mode; see the following list.

map x sequence: Define x as a sequence of editing commands.
unmap x: Disable the x definition.
map: List the characters that are currently mapped.

As with other ex-mode commands, these map commands can be saved in your .exrc file (Section 17.30) or typed in after a colon (:). If you want a keymap to use just during this editing session, you might find that vi @-functions (Section 18.4) are easier to create and use. The map commands are best for keymaps that you save in your .exrc file and use during many editing sessions.

Before you can start creating your own maps, you need to know the keys not used in command mode that are available for user-defined commands. Here's a list of the unused keys in original vi:

Letters: g K q V v
Control keys: ^A ^K ^O ^T ^W ^X
Symbols: _ * \ =

The = is used by vi if Lisp mode is set. In addition, other letters such as v may already be used in other systems.

With maps you can create simple or complex command sequences. As a simple example, you could define a command to reverse the order of words. In vi, with the cursor as shown:

you can the scroll page

the sequence to put the after scroll would be dwwP: (delete word), dw; (move to the next word), w; (put the deleted word before that word),

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Back in the old days, a terminal's arrow keys didn't work during vi text-input mode. To move around in the file, you pressed ESC and used command-mode commands like 5k and 4w. Since then, lots of vendors and users have modified vi so that you can use arrow keys during text-input mode. These days, most people think the new-fangled way that vi works is the right way. Here are some reasons to leave the arrow keys alone and do it the old way instead:

In most cases, the u (undo) command will be useless after text-input mode because the arrow keymap does several hidden commands — and u can only undo the single previous command. The only "undo" command that will do much good is U — it undoes all changes on the current line, and it probably won't work if you've moved off the line since you made the change you want to undo.
Beginners can get confused by this. They need to learn that vi is a moded editor — that you enter text in text-input mode and make changes in command mode. Movement through the file is with commands.
When people start using vi and they find that some motion commands (the cursor keys) work in text-input mode, vi seems inconsistent.
If your map! runs commands that start with an ESC (and it almost always will), your ESC key may work more slowly. That's because every time you press the ESC key, vi will wait one second (or so) to be sure that the ESC is just an ESC alone and not the beginning of a map! sequence. Some versions have changed this, though.
The fast alternative is to press ESC twice. That rings the terminal bell, though.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The vi map command (Section 18.2) lets you define keymaps: short names for a series of one or more commands. You can enter :map to define a keymap while you're editing a file with vi. But if you make a mistake, you usually have to re-enter the whole :map command to correct the problem.

@-functions (pronounced "at-functions") give you another way to define complex commands. You can define 26 @-functions named @a through @z. They're stored in named buffers (Section 17.4). So if you're also using named buffers for copying and pasting text, you'll need to share them with your @-functions.

To define an @-function:

Enter the command(s) you want to execute onto one or more lines of the file you're editing.
Yank or delete the line(s) into a named buffer with a command like "ay$ or "bD.
To use the function, type a command like @a or @b. You can repeat the function by typing @@ or a dot (.). Use u or U to undo the effects of the @-function.

Here's an example. You're editing a long HTML file with lines like these:

<STRONG>Some heading here</STRONG>

<STRONG>Another heading here</STRONG>

When you see one of those lines, you need to change the STRONGs to either H3 or H4. A global substitution with :%s won't do the job because some lines need H3 and others need H4; you have to decide line-by-line as you work through the file. So you define the function @a to change a line to H3, and @b to change to H4.

To design an @-function, start by thinking how you'd make the changes by hand. You'd probably move to the start of the line with 0, move to the right one character with l, type cw to change the word STRONG, and type in H3 (or H4). Then you'd press ESC to return to command mode. After going to the end of the line with $, you'd move to the character after the slash with T/, then change the second STRONG the way you fixed the first one.

To define the function, open a new empty line of your file (first go into text-input mode). Then type the keystrokes that will make the H3 changes; type CTRL-v before each ESC or RETURN

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I usually run vi inside windows on a system like X or the Macintosh. The window systems can copy and paste text between windows. Pasting into a vi window may be tricky if you use vi options like wrapmargin or autoindent because the text you paste can be rearranged or indented in weird ways.

I've fixed that with the upcoming keymaps. If I'm pasting in text that should be copied exactly with no changes, I go into text-input mode and type CTRL-x. That shuts off autoindent (noai) and the wrapmargin (wm=0). When I'm done pasting, I type CTRL-n while I'm still in text-input mode.

A different kind of "pasted" input is with CTRL-r. It starts the fmt (Section 21.2) utility to reformat and clean up lines while I'm pasting them. To use it, go to text-input mode and type CTRL-r. Then paste the text -- fmt will read it but not display it. Press RETURN, then CTRL-d to end the standard input to fmt. The reformatted text will be read into your vi buffer.

^[ Section 18.6

Go to http://examples.oreilly.com/upt3 for more information on: exrc

" Set 'exact' input mode for pasting exactly what is entered:

map! ^X ^[:se noai wm=0^Ma

" Set 'normal' input mode with usual autoindent and wrapmargin:

map! ^N ^[:se ai wm=8^Ma

" Read pasted text, clean up lines with fmt. Type CTRL-d when done:

map! ^R ^[:r!fmt^M

Note that some window systems convert TAB characters to spaces when you copy and paste. If you want the TABs back, try a filter-through (Section 17.18) with unexpand.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Note that when defining a map, you cannot simply type certain keys — such as RETURN, ESC, BACKSPACE, and DELETE — as part of the command to be mapped, because these keys already have meaning within ex. If you want to include one of these keys as part of the command sequence, you must escape the normal meaning by preceding the key with ^V ( CTRL-v). After CTRL-v, a carriage return appears as ^M, escape as ^[, backspace as ^H, and so on.

On the other hand, if you want to map a control character, in most cases you can just hold down the CTRL key and press the letter key at the same time. For example, to map ^A (CTRL-a), simply type:

:map CTRL-a sequence

There are, however, a few other control characters that must be escaped with a ^V. One is ^T. The others are as follows:

The characters that your account uses for erasing parts of the input you type at a command line: ^W for erasing words and ^U for erasing lines.
The characters for interrupting jobs (Section 24.11) and stopping jobs (Section 23.1).

So, if you want to map ^T, you must type:

:map CTRL-v CTRL-t sequence

The use of CTRL-v applies to any ex command, not just a map command. This means that you can type a carriage return in an abbreviation (Section 17.23) or a substitution command. For example, the abbreviation:


               :ab 123 one^Mtwo^Mthree

expands to this:

one

two

three

(The sequence CTRL-v RETURN is shown as it appears on your screen, ^M.)

You can also add lines globally at certain locations. The command:


               :g/^Section/s//As you recall, in^M&/

inserts a phrase on a separate line before any line beginning with the word Section. The & restores the search pattern.

The vertical bar (|) is used to separate multiple ex commands; it's especially difficult to quote. Because a map is interpreted when it's stored and again when it's used, you need enough CTRL-v characters to protect the vertical bar from each interpretation. You also need to protect stored CTRL-v characters by adding a CTRL-v before each one! The worst case is a text-input mode map (

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Another way to do this is with @-functions (Section 18.4).

Not every keymap is something you want to save in your .exrc file. Some maps are handy just because you have to do a repetitive series of edits. Developing a complex map to repeat your edits can save more time than it takes. For example, assume that you have a glossary with entries like this, separated by blank lines:

map - an ex command which allows you to associate

a complex command sequence with a single key.

You want to convert this glossary list to HTML format, so that it looks like:

<DT>map</DT>

<DD>

An ex command which allows you to associate

a complex command sequence with a single key.

</DD>

The best way to define a complex map is to do the edit once manually, writing down each keystroke that you must type. Then recreate these keystrokes as a map.

Use I to insert the tag for an data list term (<DT>) at the beginning of the line.
Press ESC to terminate text-input mode. Move just before the dash (t-). Use 3s to replace the dash and space after it with the closing term tag (</DT>).
Still in text-input mode, press RETURN to insert a new line. (This moves the definition to a newline underneath the <DT> tags.) Enter the opening data list definition (<DD>) tag, and press RETURN again. (The definition moves to yet another newline underneath the <DD> tag.)
Press ESC to terminate text-input mode. Your cursor is at the start of the definition. Capitalize the first word of the definition (~).
Go to the blank line after the definition (}), open a newline above (O), and insert the closing data list definition (</DD>) tag. Press ESC to end text-input mode.
Press RETURN to end the keymap definiton.

That's quite an editing chore if you have to repeat it more than a few times. With map you can save the entire sequence so that it can be re-executed with a single keystroke:

map g I<DT>^[t-3s</DT>^M<DD>^M^[~}O</DD>^[

(To store a map during a vi session, type a colon (:) first.) Note that you have to "quote" both the ESC and RETURN characters with CTRL-v (Section 18.6). ^[ is the sequence that appears when you type

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The examples that follow will give you an idea of the clever shortcuts possible when defining keyboard maps:

Add text whenever you move to the end of a word:
```
map e ea
```
Most of the time, the only reason you want to move to the end of a word is to add text. This map sequence puts you in text-input mode automatically. Note that the mapped key, e, has meaning in vi. You're allowed to map a key that is already used by vi, but the key's normal function will be unavailable as long as the map is in effect. This isn't so bad in this case, since the E command is often identical to e.
In the remaining examples, we assume that e has been mapped to ea.
Save a file and edit the next one in a series (Section 17.3):
```
map q :w^M:n^M
```
Notice that you can map keys to ex commands, but be sure to finish each ex command with a RETURN. This sequence makes it easy to move from one file to the next, and it's useful when you've opened many short files with one vi command. Mapping the letter q helps you remember that the sequence is similar to a "quit."
Put HTML emboldening codes (<STRONG> and </STRONG>) around a word:
```
map v i<STRONG>^[e</STRONG>^[
```
This sequence assumes that the cursor is at the beginning of the word. First, you enter text-input mode, then you type the code for bold font. Next, you return to command mode by typing a "quoted" (Section 18.6) ESC. Finally, you append the closing HTML tag at the end of the word, and you return to command mode. Of course, the map is not limited to HTML font tags. You can use it to enclose a word in parentheses or C comment characters, to name just a few applications.
This example shows you that map sequences are allowed to contain other map commands (the e is already mapped to ea). The ability to use nested map sequences is controlled by vi's remap option (Section 18.12), which is normally enabled.
Put HTML emboldening tags around a word, even when the cursor is not at the beginning of the word:
```
map V lbi<STRONG>^[e</STRONG>^[
```
This sequence is the same as the previous one, except that it uses

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The vi (actually, ex) command map (Section 18.2) lets you build custom vi commands. For example, the following keymap redefines the -key to run the vi commands o (open a newline below), ESCAPE, 72a- (add 72 dashes), and ESCAPE again:


               :map - o^[72a-^[

So typing - draws a row of dashes below the current line. The problem is that on versions of vi I've tried, you can't add a repetition number — that is, you can't type the command 10- to add 10 dashed lines.

The workaround is to define another macro that calls the first macro ten times. For example, to make the v key draw ten rows of dashes:


               :map v ----------

(Ugly, eh? But it works.) You might want to put the - map in your .exrc file and define "multimaps" like v while you're running vi.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You may want to input text in all uppercase letters. Using CAPS LOCK in vi can be a pain because you have to release CAPS LOCK almost every time you want to type a vi command. Here's a nice way to type lowercase letters during input and ex modes; they'll be mapped to uppercase automatically.

Try putting this in your .exrc (Section 17.5) file:

map! a A

map! b B

map! c C

  ...

map! z Z

Anytime you type (during text-input mode) an a, the editor will map it into A. What's that you say . . . you don't want this all the time? Just put it in a file called .f (for FORTRAN), and type:


               :source .f

when you want FORTRAN mode. Of course, you can define a function key (Section 18.2) to :source this.

[After that, anywhere you want a lowercase letter, type CTRL-v first to cancel the map temporarily. For example, to type the command :w, type : CTRL-v w.

You can also go into the ex command mode by typing the vi command Q. That takes you to the ex colon (:) prompt — where the map! macros won't affect what you type. To return to vi mode from ex command mode, type :vi. -- JP]

—BB, in net.unix on Usenet, 9 October 1986

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some people don't like to press ESC first to move the cursor while they're using vi. These keymaps change CTRL-h, CTRL-j, CTRL-k, and CTRL-l to do the same things during input mode as the commands h, j, k, and l do in command mode.

Is your erase character set to CTRL-h (Section 5.8) outside vi? If it is, mapping CTRL-h (usually labeled BACKSPACE on your keyboard) will change the way CTRL-h works during text-input mode: instead of erasing the characters you've typed since you entered text-input mode, now CTRL-h will move backwards over what you type without erasing it. One workaround is to change your Unix erase character to the DELETE or RUBOUT key by typing the command stty erase '^?' before you start vi. Then your DELETE key will erase what you type, and the BACKSPACE key will jump back over it without deleting.

The lines for your .exrc file (Section 17.30) are as follows:

Go to http://examples.oreilly.com/upt3 for more information on: exrc

map! ^H ^[i

map! ^K ^[ka

map! ^L ^[la

map! ^V

 ^[ja

" Note: the two lines above map ^J (LINEFEED)

That last map takes two lines; it's tricky and may not work right on all versions of vi. No, it isn't a map for CTRL-v, though that's what it looks like. It maps ^J, the LINEFEED key. The ^V comes at the very end of its line. When you're entering that keymap, type CTRL-v and then press LINEFEED or CTRL-j. The cursor will move to the start of the next line; type a SPACE and the rest of the macro. It's a good idea to add the reminder comment (starting with the comment character, a double quote (")), on the line below the map.

This map for CTRL-j is obviously something for which the people who wrote my version of vi didn't plan. For example, look at the mess it makes when I ask for a list of my text-input keymaps:

:map!^H      ^H      ^[i

^K      ^K      ^[ka

^L      ^L      ^[la



        ^[ja

Before you use this map on important files, you probably should test it carefully.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

For years, I assumed that I could map (Section 18.2) only a few keys in vi — the characters like v and ^A that aren't used. For instance, if I mapped ^F to do something else, I thought I'd lose that handy "forward one screen" command. You thought the same thing? Then we're both wrong!

Just use the noremap option. Here's a simple example. You can make ^F the "show file information" (normally ^G ) command. Then, make ^A take over the "forward (ahead) one screen" function. Put these lines in your .exrc file (Section 17.5):

set noremap

map ^F ^G

map ^A ^F

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you add text to the start of a line and make the line longer than your screen's width, vi won't break ("wrap") the line unless your cursor crosses the wrapmargin point near the righthand edge of the screen. You can get lines that are too long.

Here are two macros that cut (Kut) the current line:

Go to http://examples.oreilly.com/upt3 for more information on: exrc

map K 78^V|lBhr^M

map K 078lF r^M

The first macro doesn't seem to work on some versions of vi. It's the better one though, because it uses the | (vertical bar) command to move to column 78, even if there are TABs in the line. Then it moves one more character to the right (if it can), moves back to the start of the word, moves back one more character onto the blank or TAB before the word, and replaces that character with a RETURN.

The second macro counts TABs as single characters, but it works on every version of vi I've tried. It moves to the left edge, then to the 79th character, then back to the previous space. Finally, it replaces that space with a carriage return.

You might try adding a J to the start of either macro. That'll join the next line to the current one before cutting; it might make a nicer "wrap." Another way to do this is with a filter-through (Section 17.18) and the fmt (Section 17.28) command:


               !!fmt

That will break the current line neatly, though it also might change the spacing after periods (.) or replace leading TABs with spaces.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Emacs automatically keeps backup copies of the file you're editing. If you have editing problems (or just change your mind), you can get the previous file version by recovering from a backup file. I like this idea, but I don't like the way that backups are done automatically. Instead, I want to choose when vi makes a backup "snapshot." This macro, CTRL-w, lets me do that: it writes a copy of the current filename as filename~. (The trailing tilde (~) is an Emacs convention. Section 14.17 shows ways to remove these backup files.) Whenever I want to save a snapshot of the editing buffer, I just type CTRL-w.

^M Section 18.6

map ^W :w! %~^M

The w! writes without questions, overwriting any previous backup with that name. vi replaces % (percent sign) with the filename (or pathname) you're currently editing.

If you want an Emacs-style backup to be made every time you write the file (except the first time), you could try something like this:

map ^W :!cp -pf % %~^M:w^M

The first command uses cp -p (Section 10.12) to make a backup of the previously written file; the cp -f option forces the write. (vi may warn you

File modified

since

last write, but the versions I've checked will run cp anyway.) The next command writes the current editing buffer into the file.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The "other" interactive editor that's commonly used is Emacs. Emacs actually refers to a family of editors; versions of Emacs run under most operating systems available. However, the most important (and most commonly used) version of Emacs is "GNU Emacs," developed by the Free Software Foundation.

Go to http://examples.oreilly.com/upt3 for more information on: emacs

GNU Emacs is popular because it's the most powerful editor in the Emacs family; it is also freely available under the terms of the FSF's General Public License. Although there are certainly strong differences of opinion between Emacs and vi users, most people agree that Emacs provides a much more powerful and richer working environment.

What's so good about Emacs, aside from the fact that it's free? There are any number of individual features that I could mention. (I'll give a list of favorite features in Section 19.2.) Emacs' best feature is the extent to which it interacts with other Unix features. For example, it has a built-in email system so you can send and receive mail without leaving the editor. It has tools for "editing" (deleting, copying, renaming) files, for running a Unix shell within Emacs, and so on. The C shell has a rather awkward command-history mechanism; the Korn shell has something more elaborate. But imagine being able to recall and edit your commands as easily as you edit a letter! That's far beyond the abilities of any shell, but it's simple when you run a shell inside your editor.

In this book, we can't give anywhere near as much attention to Emacs as we can to vi (Section 17.1), but we will point out some of its best features and a few tricks that will help you get the most out of it. For the impatient, here's a very brief survival guide to Emacs.

Starting Emacs: Like vi, Emacs can be started from the shell prompt by typing its name, emacs. Once started, emacs will present you with a helpful screen of commands. A word of advice: take the tutorial (CTRL-h t). If you want to edit an existing file, simply type

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The "other" interactive editor that's commonly used is Emacs. Emacs actually refers to a family of editors; versions of Emacs run under most operating systems available. However, the most important (and most commonly used) version of Emacs is "GNU Emacs," developed by the Free Software Foundation.

Go to http://examples.oreilly.com/upt3 for more information on: emacs

GNU Emacs is popular because it's the most powerful editor in the Emacs family; it is also freely available under the terms of the FSF's General Public License. Although there are certainly strong differences of opinion between Emacs and vi users, most people agree that Emacs provides a much more powerful and richer working environment.

What's so good about Emacs, aside from the fact that it's free? There are any number of individual features that I could mention. (I'll give a list of favorite features in Section 19.2.) Emacs' best feature is the extent to which it interacts with other Unix features. For example, it has a built-in email system so you can send and receive mail without leaving the editor. It has tools for "editing" (deleting, copying, renaming) files, for running a Unix shell within Emacs, and so on. The C shell has a rather awkward command-history mechanism; the Korn shell has something more elaborate. But imagine being able to recall and edit your commands as easily as you edit a letter! That's far beyond the abilities of any shell, but it's simple when you run a shell inside your editor.

In this book, we can't give anywhere near as much attention to Emacs as we can to vi (Section 17.1), but we will point out some of its best features and a few tricks that will help you get the most out of it. For the impatient, here's a very brief survival guide to Emacs.

Starting Emacs: Like vi, Emacs can be started from the shell prompt by typing its name, emacs. Once started, emacs will present you with a helpful screen of commands. A word of advice: take the tutorial (CTRL-h t). If you want to edit an existing file, simply type emacs with the desired filename after it. While editing your file, you may save your work to disk with CTRL-x CTRL-s.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's the list we promised — a list of our favorite features:

Windows: Emacs is a "windowed editor." Before anyone heard of the X Window System or the Macintosh, Emacs had the ability to divide a terminal's screen into several "windows," allowing you to do different things in each one. You can edit a different file in each window or read mail in one window, answer mail in another, issue shell commands in a third, and so on.
Now that we all have nice workstations with mice and other crawly things for navigating around a bitmapped screen, why do you care? First, you may not have a bitmapped screen, and even if you have one in the office, you may not at home. Second, I still find Emacs preferable to most "modern" window systems because I don't have to use a mouse. If I want to create another window, I just type CTRL-x 2 (which splits the current window, whatever it is, into two); if I want to work in another window, I just type CTRL-x o; if I want to delete a window, I type CTRL-x 0. Is this faster than reaching for my mouse and moving it around? You bet. Particularly since my mouse is hidden under a pile of paper. (Of course, it's hidden because I hardly ever need it.) Once you've created a window, it's just as easy to start editing a new file, initiate a shell session, and so on. Third, even though you're using a windowing system, you may not have a lot of screen real estate available. By having a split Emacs screen, all editing can be done in one window, leaving enough room for other applications, such as the Mozilla web browser, to be open. Whether you're developing web pages or just reading Slashdot while "working," you'll appreciate the free space on the screen. It isn't uncommon for Emacs users to always have Emacs open on their desktops.
Shells: You can start an interactive shell within any Emacs window; just type ESC-x shell, and you'll see your familiar shell prompt. It's easy to see why this is so useful. It's trivial to return to earlier comands, copy them, and edit them. Even more important, you can easily take the output of a command and copy it into a text file that you're editing — obviously an extremely useful feature when you're writing a book like this. Emacs also lets you issue commands that operate on the contents of a window or a selected region within a window. Another benefit to doing shell work directly in Emacs is that every word that appears in that shell buffer is available for

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Emacs customizations are usually stored in a file called .emacs in your home directory. In Section 19.7, we've given a few customizations that I personally find convenient; if you're like most people, you'll add customizations over time. You'll end up doing this even if you're not a LISP programmer; if you know any other Emacs users, you'll soon be borrowing their shortcuts. The best way to customize Emacs to your taste is to find out what works for others . . . and then steal it. For that matter, many — if not most — of the customizations in my file were stolen from other users over the years. I hope I've gotten this process off to a good start.

However, you should also be aware of the "dark side" of customization. What happens if you sit down at someone else's system, start Emacs, and find out that he's customized it so extensively that it's unrecognizable? Or that a "helpful" administrator has installed some system-wide hacks that are getting in your way? Here's what will help. First, start emacs with the option -q; that tells Emacs not to load any .emacs initialization file. (If you want to load your initialization file instead of someone else's, try the option -u username).

That still doesn't solve the problem of system-wide customizations. To keep those from getting in the way, put the following line at the beginning of your .emacs file:

(setq inhibit-default-init t)

This turns off all "global" initializations. (If you're sharing someone else's system, you may still need the -u option to force Emacs to read your initialization file.)

—ML, DC, and BR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're like most people, you often spend a few hours editing a file, only to decide that you liked your original version better. Or you press some strange sequence of keys that makes Emacs do something extremely weird and that you can't "undo." Emacs provides several ways to get out of these tight spots.

First, try the command ESC-x revert-buffer. Emacs will ask one of two questions: either "Buffer has been auto-saved recently. Revert from auto-save file? (y or n)" or "Revert buffer from file your-filename? (yes or no)".

Before deciding what to do, it's important to understand the difference between these two questions. Emacs creates an auto-save file every 300 keystrokes you type. So, if you're reverting to the auto-save file, you'll at most lose your last 300 keystrokes. Maybe this is what you want — but maybe you made the mistake a long time ago. In that case, you don't want to use the auto-save file; type n, and you'll see the second question, asking if you want to revert to the last copy of the file that you saved. Type yes to go back to your most recent saved version.

It's possible that you'll only see the second question ("Revert buffer from file . . . "). This means that you have saved the file sometime within the last 300 keystrokes. As soon as you save a file, Emacs deletes the auto-save file. It will create a new one every 300 keystrokes.

It's worth noting that Emacs is very picky about what you type. If it asks for a y or an n, you've got to type y or n. If it asks for yes or no, you've got to type yes or no. In situations like this, where the two styles are mixed up, you've got to get it right.

If you're in real trouble and you want to go back to your original file — the way it was when you started editing — you need to recover Emacs' backup file. If you're editing a file that already exists, Emacs will create a backup file as soon as it starts. If you're editing a new file, Emacs will create a backup the second time you save the file. Once it's created, the backup file is never touched; it stays there until the next time you start Emacs, at which point you'll get a new backup, reflecting the file's contents at the start of your editing session.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many users are used to editors that are normally in overwrite mode: when you backspace and start typing, you type over the character that is underneath the cursor. By default, Emacs works in insertion mode, where new characters are inserted just before the cursor's position.

If you prefer overwrite mode, just give the command ESC-x overwrite-mode. You can use command abbreviation (Section 19.6) to shorten this to ESC-x ov. On many keyboards, pressing INSERT also turns on overwrite mode. If you get tired of overwrite mode, use the same command to turn it off.

If you always want to use overwrite mode, create a file named .emacs in your home directory, and put the following line in it:

(setq-default overwrite-mode t)

This is a simple Emacs customization; for a lot more about customization, see O'Reilly & Associates' Learning GNU Emacs, by Bill Rosenblatt, Eric Raymond, and Debra Cameron.

—ML and DC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Emacs has a great feature called command completion. Basically, command completion means that Emacs will let you type the absolute minimum and it will fill in the rest. You can use command completion whenever you're typing a filename, buffer name, command name, or variable name. Simply type enough of the name to be "unique" (usually the first few letters), followed by a TAB. Emacs will fill in the rest of the name for you. If the name isn't unique — that is, if there are other filenames that start with the same letters — Emacs will show you the alternatives. Type a few more letters to select the file you want, then press TAB again.

For example, if I'm trying to load the file outline.txt, I can simply give the command CTRL-x CTRL-f out TAB. Providing that there are no other filenames beginning with the letters out, Emacs will fill in the rest of the filename. When I see that it's correct, I press RETURN, and I'm done.

When you use command completion, always make sure that Emacs has successfully found the file you want. If you don't, the results may be strange: you may end up with a partial filename or the wrong file.

Along the same lines as command completion is a feature called dynamic expansion . After typing the first few letters of a word, you can have Emacs search all open buffers for completions of that word. Simply type ESC-/, and emacs will complete the partial word with one you've already typed. You can cycle through all the choices by repeating the keystroke. Warning: this feature is addictive.

—ML and BR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I'm a very fast typist — which means that I hate using special function keys, arrow keys, and especially mice. I deeply resent anything that moves me away from the basic alphanumeric keyboard. Even BACKSPACE and DELETE are obnoxious, since they force me to shift my hand position.

With this in mind, I've customized Emacs so that I can do virtually anything with the basic alphabetic keys, plus the CONTROL key. Here are some extracts from my .emacs file:

Go to http://examples.oreilly.com/upt3 for more information on: .emacs_ml

;; Make CTRL-h delete the previous character. Normally, this gets

;; you into the "help" system.

 (define-key global-map "\C-h" 'backward-delete-char)

;; make sure CTRL-h works in searches, too

 (setq search-delete-char (string-to-char "\C-h"))

;; bind the "help" facility somewhere else (CTRL-underscore).

;; NOTE: CTRL-underscore is not defined on some terminals.

 (define-key global-map "\C-_" 'help-command) ;; replacement

 (setq help-char (string-to-char "\C-_"))

;; Make ESC-h delete the previous word.

 (define-key global-map "\M-h" 'backward-kill-word)

;; Make CTRL-x CTRL-u the "undo" command; this is better than "CTRL-x u"

;; because you don't have to release the CTRL key.

 (define-key global-map "\C-x\C-u" 'undo)

;; scroll the screen "up" or "down" one line with CTRL-z and ESC z

 (defun scroll-up-one ( ) "Scroll up 1 line." (interactive)

   (scroll-up (prefix-numeric-value current-prefix-arg)))

 (defun scroll-down-one ( ) "Scroll down 1 line." (interactive)

   (scroll-down (prefix-numeric-value current-prefix-arg)))

 (define-key global-map "\C-z" 'scroll-up-one)

 (define-key global-map "\M-z" 'scroll-down-one)

;; Use CTRL-x CTRL-v to "visit" a new file, keeping the current file

;; on the screen

 (define-key global-map "\C-x\C-v" 'find-file-other-window)

The comments (lines beginning with two semicolons) should adequately explain what these commands do. Figure out which you need, and add them to your .emacs file. The most important commands are at the top of the file.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Emacs has, oh, a hundred or so different search commands. (Well, the number's probably more like 32, but who's counting?) There are searches of absolutely every flavor you could ever imagine: incremental searches, word searches, regular-expression searches, and so on.

However, when it comes to your plain old garden-variety search, Emacs is strangely deficient. There is a simple search that just looks for some arbitrary sequence of characters, but it's rather well hidden. In addition, it lacks one very important feature: you can't search for the same string repeatedly. That is, you can't say "Okay, you found the right sequence of letters; give me the next occurrence"; you have to retype your search string every time.

Go to http://examples.oreilly.com/upt3 for more information on: search.el

I thought this was an incredible pain until a friend of mine wrote a special search command. It's in the file search.el. Just stick this into your directory for Emacs hacks (Section 19.12), and add something like the following to your .emacs file:

;; real searches, courtesy of Chris Genly

;; substitute your own Emacs hack directory for /home/los/mikel/emacs

 (load-file "/home/los/mikel/emacs/search.el")

Now you can type CTRL-s to search forward and CTRL-r to search back. Emacs will prompt you for a search string and start searching when you press RETURN. Typing another CTRL-s or CTRL-r repeats your previous search. When you try this, you'll see one other useful feature: unlike the other Emacs searches, this kind of search displays the "default" (i.e., most recent) search string in the minibuffer. It's exactly the kind of search I want.

It's conceivable that you'll occasionally want incremental searches. You'll have to "rebind" them, though, to use them conveniently. Here are the key bindings that I use:

;; rebind incremental search as ESC-s and ESC-r

 (define-key global-map "\M-s" 'isearch-forward)

 (define-key global-map "\M-r" 'isearch-backward)

 ;; have to rebind ESC s separately for text-mode. It's normally

 ;; bound to 'center-line'.

 (define-key text-mode-map "\M-s" 'isearch-forward)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I've seen a number of strange situations in which Emacs can't find files unless you type a complete ("absolute") pathname ( Section 1.16), starting from the root (/ ). When you try to visit a file, you'll get the message File not found and directory doesn't exist.

In my experience, this usually means that the C shell's PWD environment variable (Section 35.5) has been incorrectly set. There are a few (relatively pathological) ways of tricking the C shell into making a mistake. More commonly, though, I've seen a few systems on which the C shell sticks an extra slash into PWD: that is, its value will be something like /home/mike//Mail rather than /home/mike/Mail. Unix doesn't care; it lets you stack up extra slashes without any trouble. But Emacs interprets // as the root directory — that is, it discards everything to the left of the double slash. So if you're trying to edit the file /home/mike//Mail/output.txt, Emacs will look for /Mail/output.txt. Even if this file exists, it's not what you want. [This also happens when Emacs is called from a (Bourne) shell script that has changed its current directory without changing PWD. — JP]

This problem is particularly annoying because the shell will automatically reset PWD every time you change directories. The obvious solution, sticking

unsetenv

PWD

in your .cshrc file, doesn't do any good.

What will work is defining an alias (Section 29.1):

(..) Section 43.7

alias gmacs "(unsetenv PWD; emacs \!*)"

A better solution might be to switch to another shell that doesn't have this problem. The Bourne shell (sh) obviously doesn't, since it doesn't keep track of your current directory.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I remember being driven absolutely crazy by a guy (who hopefully won't be reading this) who called me every other week and asked me how to stick a page break into some text file he was writing. He was only printing on a garden-variety printer, for which inserting a page break is a simple matter: just add a formfeed character, CTRL-l. But CTRL-l already means something to Emacs ("redraw the screen"). How do you get the character into your file, without Emacs thinking that you're typing a command?

Simple. Precede CTRL-l with the "quoting" command, CTRL-q. CTRL-q tells Emacs that the next character you type is text, not a part of some command. So the sequence CTRL-q CTRL-l inserts the character CTRL-l into your file; you'll see ^L on your screen. (Note that this represents a single character, instead of two characters.) In turn, when you print the file on many printers, the CTRL-l will cause a page eject at the appropriate point.

You can use this technique to get any "control character" into an Emacs file. In fact, under pressure I've done some pretty bizarre binary editing — not a task I'd recommend, but certainly one that's possible.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Like vi, Emacs provides an "abbreviation" facility. Its traditional usage lets you define abbreviations for long words or phrases so you don't have to type them in their entirety. For example, let's say you are writing a contract that repeatedly references the National Institute of Standards and Technology. Rather than typing the full name, you can define the abbreviation nist. Emacs inserts the full name whenever you type nist, followed by a space or punctuation mark. Emacs watches for you to type an abbreviation, then expands it automatically as soon as you press the spacebar or type a punctuation mark (such as ., ,, !, ?, ;, or :).

One use for word-abbreviation mode is to correct misspellings as you type. Almost everyone has a dozen or so words that he habitually types incorrectly, due to some worn neural pathways. You can simply tell Emacs that these misspellings are "abbreviations" for the correct versions, and Emacs fixes the misspellings every time you type them. If you take time to define your common typos as abbreviations, you'll never be bothered with teh, adn, and recieve when you run the spellchecker. Emacs sweeps up after your typos and corrects them. For example, let's say that you define teh as an abbreviation for the. When you press the spacebar after you type teh, Emacs fixes it immediately, and you continue happily typing. You may not even notice that you typed the word wrong before Emacs fixes it.

Usually, if you go to the trouble of defining a word abbreviation, you will use it in more than one Emacs session. But if you'd like to try out abbreviation mode to see if you want to make it part of your startup, use the following procedure to define word abbreviations for this session:

Enter word-abbreviation mode by typing ESC-x abbrev-mode. abbrev appears on the mode line.
Type the abbreviation you want to use, and press CTRL-x a. Emacs then asks you for the expansion.
Type the definition for the abbreviation, and press RETURN. Emacs then expands the abbreviation; it will do so each time you type it followed by a space or punctuation mark. The abbreviations you've defined will work only during this Emacs session.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you use any Emacs editor (GNU Emacs or any of the commercial alternatives), you may have written lots of handy LISP programs to use while you're editing. It's convenient to create a separate directory for these; a good name for this directory is (obviously) emacs; it's usually located in your home directory.

If you use GNU Emacs, you should put the following line in the .emacs file:

(setq load-path (append load-path '("your-emacs-directory")))

This tells Emacs that it should look in your personal Emacs directory to find your programs. (It's similar to the PATH (Section 35.6) environment variable.)

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you have time to waste (and I mean really waste), Emacs has things to keep you occupied. There are lots of interesting special effects, such as "disassociated text." My favorite is a command called "psychoanalyze-pinhead." It's based on "doctor," which is a variation of the classic artificial-intelligence demonstration called "Eliza": the computer plays psychotherapist and asks you questions; you reply, and your answers are the basis for new questions. Nothing new, really; I remember it from my high school days, in the early 1970s. If you think you might be in need of a psychotherapist, save your money and try ESC-x doctor RETURN.

If you want to see something really interesting, try ESC-x psychoanalyze-pinhead. This takes the built-in "doctor" program and feeds it with quotations from the cult comic strip Zippy the Pinhead. The result is indeed bizarre. Here's some typical output:

I am the psychotherapist.  Please, describe your problems.  Each time

you are finished talking, type RET twice.



YOU PICKED KARL MALDEN'S NOSE!!



Why do you say I picked karl malden's nose?



Am I SHOPLIFTING?



Is it because of your plans that you say are you shoplifting?



Are you selling NYLON OIL WELLS??  If so, we can use TWO DOZEN!!

Press CTRL-c when you've had enough.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In the "old days," when programmers worked on printing terminals, editing was done one line at a time. Editors that let you move a cursor around the screen to select text to edit hadn't yet been invented, because there weren't any screens to look at text on!

With ever more advanced WYSIWYG (What You See Is What You Get) word processors and editing programs, it's easy for novices to think of line editors as a bizarre relic. Perhaps they are — but if so, they are a relic of extraordinary power.

You see, line editors lend themselves to scripting — the ability to write what in effect are editing programs that can be applied over and over to different files.

When we talk about "batch editing" or scripts, here are some of the programs you might use:

ed is the original Unix line editor.
ex supports a superset of ed commands; it is widely used from within vi, which is the ex "visual" or "screen" mode.
sed ( Section 34.1) is an editor that can only be run with scripts or by entering a few short commands as command-line arguments; while it has many similar commands, it has some important differences (Section 34.2) from ed and ex.
awk ( Section 20.10) is a great way to pull apart a line of text into a sequence of elements. Used frequently with sed.
patch ( Section 20.9) is a specialized editor designed to apply editing scripts created with diff (Section 11.1). You can do this with ed or ex as well, but patch is especially clever at it.

Of course, editing is a continuum, and beyond sed and awk (Section 20.10) lie more complete programming languages like perl (Section 41.1) and python (Section 42.1) that are very adept at manipulating text.

— TOR

When you write a script that contains a series of editing actions and then run the script on an input file, you take what would be a hands-on procedure in an editor such as

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In the "old days," when programmers worked on printing terminals, editing was done one line at a time. Editors that let you move a cursor around the screen to select text to edit hadn't yet been invented, because there weren't any screens to look at text on!

With ever more advanced WYSIWYG (What You See Is What You Get) word processors and editing programs, it's easy for novices to think of line editors as a bizarre relic. Perhaps they are — but if so, they are a relic of extraordinary power.

You see, line editors lend themselves to scripting — the ability to write what in effect are editing programs that can be applied over and over to different files.

When we talk about "batch editing" or scripts, here are some of the programs you might use:

ed is the original Unix line editor.
ex supports a superset of ed commands; it is widely used from within vi, which is the ex "visual" or "screen" mode.
sed ( Section 34.1) is an editor that can only be run with scripts or by entering a few short commands as command-line arguments; while it has many similar commands, it has some important differences (Section 34.2) from ed and ex.
awk ( Section 20.10) is a great way to pull apart a line of text into a sequence of elements. Used frequently with sed.
patch ( Section 20.9) is a specialized editor designed to apply editing scripts created with diff (Section 11.1). You can do this with ed or ex as well, but patch is especially clever at it.

Of course, editing is a continuum, and beyond sed and awk (Section 20.10) lie more complete programming languages like perl (Section 41.1) and python (Section 42.1) that are very adept at manipulating text.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you write a script that contains a series of editing actions and then run the script on an input file, you take what would be a hands-on procedure in an editor such as vi and transform it into a look-no-hands procedure.

When performing edits manually, you get to trust the cause-and-effect relationship of entering an editing command and seeing the immediate result. There is usually an "undo" command that allows you to reverse the effect of a command and return the text file to its previous state. Once you learn an interactive text editor, you have the feeling of making changes in a safe and controlled manner, one step at a time.

Most people new to "power editing" will feel there is greater risk in writing a script to perform a series of edits than in making those changes manually. The fear is that by automating the task, something will happen that cannot be reversed. The object of learning scripting with ex or sed is to understand the commands well enough to see that your results are predictable. In other words, you come to understand the cause-and-effect relationship between your editing script and the output you get.

This requires using the editor in a controlled, methodical way. Gradually, you will develop methods for creating and testing editing scripts. You will come to rely upon these methods and gain confidence that you know what your script is doing and why.

Here are a few tips:

Carefully examine your input file, using grep, before designing your script.
Start with a small sample of occurrences in a test file. Run your script on the sample and make sure the script is working. Remember, it's just as important to make sure the script doesn't work where you don't want it to. Then increase the size of the sample. Try to increase the complexity of the input.
Work carefully, testing each command that you add to a script. Compare the output against the input file to see what has changed. Prove to yourself that your script is complete. Your script may work perfectly based on your assumptions of what is in the input file, but your assumptions may be wrong.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The key to making line editors work for you is understanding how to select (or "address") the lines that will be affected by the commands in your script.

In ed and ex, a command affects only the "current" line — the first line of the file to begin with, and later the site of the last edit or movement command — unless you precede the command with an address to indicate some other line or lines. In sed, most commands apply to every line unless you give an address.

Most line editors address lines in three ways:

with line numbers
with regular expression patterns
with special symbols

It's possible to address single lines or a range of lines.

Table 20-1 describes the addresses you can use with ex.

Table 20-1: Line addressing in the ex editor
Address	Description
`1,$`	All lines in the file.
`%`	All lines; same as `1,$`.
x,y	Lines x through y.
x `;` y	Lines x through y, with current line reset to x.
`1`	Top of file.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many line editor commands are not particularly useful in scripts. The two commands that you will use far and away the most often are s (substitute), to replace one pattern with another, and d (delete), to delete one or more lines. On occasion, though, you'll want to insert text from a script. (Editing scripts built by diff (Section 18.6) make heavy use of insert, append, delete, and change commands.) And of course, you need commands to write the file and quit the editor.

Here's the syntax of most of the commands you may encounter in ex editing scripts. (The ed editor understands the abbreviated versions of some, but not all, of these commands.) Elements in [brackets] are optional; don't type the [ or ]. The leading colon (:) shown in examples is the ex command character used to issue an ex command from vi; in a script, the colon would be omitted. The autoindent feature referred to below aids anyone writing structured text. Your editor can ease the burden of creating outlines and source code by positioning the cursor beneath the first character of the previous line.

append: [ address ] a[!] text .
Append text at specified address, or at present address if none is specified. Add a ! to switch the autoindent setting that will be used during input. For example, if autoindent was enabled, ! disables it.
change: [ address ] c[!] text .
Replace the specified lines with text. Add a ! to switch the autoindent setting during input of text.
copy: [ address ] co destination

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Because vi is built on top of the ex line editor, you get all the power of a line editor as well. Any experienced vi user issues ex commands all the time — but usually one by one, at the colon (:) prompt.

The one exception is the .exrc file (Section 17.5), which is, at bottom, a list of commands for ex to run on startup — in short, an editor script.

What many beginners don't know is that you can save a sequence of ex commands in any file and execute it with the :so command (Section 20.4). For example, Bruce Barnett uses this trick to set himself up specially for editing FORTRAN programs (Section 18.10).

In general, sed ( Section 34.1) is better for general-purpose batch editing — such as making a set of global substitutions over and over again on multiple files — therefore, :so is most often used for reading in setup commands. Keep in mind, though, any time you find yourself issuing the same commands over and over again, think script!

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: diff

The diff command can make an editing script that you give to the ex or ed editors or the patch (Section 20.9) program. They'll apply your same edits to other copies of the same file. This is handy if you have a lot of copies of a big file, spread around a network or on a lot of disks, and you want to make the same changes to all the

Section 20.8

files. In fact, this is how the Concurrent Version Control (CVS) system works. Instead of sending new copies of the whole file, just have diff make a script — and use that little script to update all the big files.

Here's a demo. I'm going to modify a program called pqs.c. Then I'll use diff and ed to apply the same changes to a copy of the file named remote-pqs.c (which might be at a remote computer):

>> Section 43.1

1% cp pqs.c remote-pqs.c

2% cp pqs.c pqs.c.new

3% vi pqs.c.new

4% diff pqs.c pqs.c.new

106,107c106

<       fprintf(stderr,

<           "%s: quitting: not able to %s your .pq_profile file.\n",

--

>       fprintf(stderr, "%s: quitting: can't %s your .pq_profile file.\n",

390a390

>               "WARNING:",

5% diff -e pqs.c pqs.c.new > edscr

6% cat edscr

390a

                "WARNING:",

.

106,107c

        fprintf(stderr, "%s: quitting: can't %s your .pq_profile file.\n",

.

7% echo w >> edscr

8% ed remote-pqs.c < edscr

19176

19184

9% diff pqs.c.new remote-pqs.c

10%

At prompt 1%, I make the simulated "remote" copy of the pqs.c file. At prompt 2%, I make another copy of it; at prompt 3%, I edit the copy. Prompt 4% has a diff that shows the changes I made. Then, at prompt 5%, I run diff -e (Section 11.1); I save the result in edscr, which I show at prompt 6%.

Prompt 7% is important because diff -e doesn't add a w command to the script file. That tells ed to write its changes to the file. I use echo w (Section 27.5) to add the command.

In prompt 8%, I give ed the name of the "remote" file to edit as a command-line argument and give it the script file on its standard input. At prompt

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

What ed and ex lack in intutitive interface design, they make up for when used in batch editing shell scripts. For example, you might be maintaining a web site with a lot of static content all stored in traditional HTML files. One such file might look like this:

<html>

<body>

<h1>Hello, world!</h1>

<p>Glad you could make it

<img src="/graphics/smiley.gif" alt="[:-)]">.

<p>Here's a picture of my house:

<img src="/graphics/house.gif" alt="[my house]">

</body>

</html>

One day, you get an email that all the images will now be served out of the directory /img instead of /graphics. Also, all existing gif files have been replaced with png files. Although these changes don't sound like much, making these modifications to a large number of files quickly becomes tedious. By writing a shell script that calls either ed or ex, you will not only solve today's problem, but you'll also be ready to make new changes to the files whenever that becomes necessary. A Bourne shell script that makes these changes looks like the following:

#!/bin/sh

# Convert some of the hard coded values in HTML

# into the new site schema



# Patterns to be replaced

old_graphics_dir="graphics"

old_graphics_ext="gif"



# new values

new_graphics_dir="img"

new_graphics_ext="png"



# Make the changes

for file in *html;

do

   ed $file <<EOF

1,\$s/$old_graphics_dir/$new_graphics_dir/g

1,\$s/$old_graphics_ext/$new_graphics_ext/g

w

EOF

done

The script is fairly simple. It defines a few variables to hold the patterns to be found and replaced. The replacement values are defined next. This script is meant to be run in the directory containing all the HTML files. The list of all files ending in "html" is iterated over in a for loop in which ed is fed commands from a here document. Recall that $ is a special character for Bourne shell and must be escaped in the line-addressing part of the ed command. After the search and replace operations finish, the ed buffers need to be written back to disk with the w command. This script works with both ed and ex.

In older versions of

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

People use the ed editor with script files to make global edits. But many versions of ed can't edit large files. The ex editor is usually better, but it has limits, too. How large is "large"? That depends on your version. Most eds I've seen can't handle more than about 100,000 characters.

There are no limits on sed (Section 34.1), although you'll need to save its output somehow (Section 34.4), and your editing script may have to be changed to work with sed. Here's what you'll see when ed fails:

% cat edscr 

s/Unix/UNIX/g

w

% ed - words < edscr

?

%

The ? is ed's "verbose" way of telling you that something's wrong. This obscure message is especially bad if you write a shell script that edits multiple files in a loop; you may not notice the error or be able to tell which file had the problem. Be sure your script checks for errors!

Unfortunately for programmers, ed may not return an error status that you can test. There are workarounds, though. When the ed - command succeeds, it doesn't display anything. The simplest way to find errors is to check for any output on stdout or stderr. This chunk of a Bourne shell script shows how (your filename is in the shell variable $filename (Section 35.9)):

2>&1 Section 36.16, [ ] Section 35.26, $? Section 35.12

edout="`ed - $filename < edscr 2>&1`"

if [ -n "$edout" -o $? -ne 0 ]

then

   echo "$edout" 1>&2

   echo "QUITTING: 'ed - $filename < edscr' failed?!?" 1>&2

   exit 1

fi

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: patch

Like all of Larry Wall's widely used programs (including perl (Section 41.1), a software configuration script called Configure, and the rn news reader), patch betrays a whimsical intelligence of its own. Feed it any kind of diff listing ( Section 11.1) (not just an editing script produced with the -e option — the typical way before patch came around). patch figures out what it needs to do to apply the diff, and updates the file, supplying all the while a breezy commentary on what it's doing:

% patch < testfile.diff

Hmm...  Looks like a normal diff to me...

File to patch: testfile

Patching file testfile using Plan A...

Hunk #1 succeeded at 2.

done

As Larry once noted, patch has done an awful lot to "change the culture of computing." Almost all free software is now updated by means of patches rather than complete new releases. patch is smart enough to discard any leading or trailing garbage (such as mail headers or signatures), so a program source file can be updated by piping a mail message containing a diff listing between old and new versions directly to patch.

Here are a few of the other things patch is smart enough to do:

Figure out the name of the file to be updated and do it without asking (usually only if the diff file is a context diff (Section 11.3) produced with the -c option).
Look for a suitable RCS or CVS (Section 39.4) file and check it out, if the filename itself can't be found.
Handle diff listings that don't quite match. This makes it possible for patch to update a file that the recipient has already changed from the one that the diff was based on.
Save any pieces of the diff file that don't end up being used, in a file named by adding the suffix .rej (reject) to the name of the file being patched.
Back up the file that is being patched, appending the suffix .orig to the name of the file being patched.
Recognize that its input may actually apply to several files, and patch each of them separately. For example, a whole directory might be updated by a "patch" file that contained diff listings for each of the files in the directory. (By the way, the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Up to this point, we've shown you tools to do basic batch editing of text files. These tools, although powerful, have limitations. Although you can script ex commands, the range of text manipulation is quite limited. If you need more powerful and flexible batch editing tools, you need to look at programming languages that are designed for text manipulation. One of the earliest Unix languages to do this is awk, created by Al Aho, Peter Weinberger, and Brian Kernighan. Even if you've never programmed before, there are some simple but powerful ways that you can use awk. Whenever you have a text file that's arranged in columns from which you need to extract data, awk should come to mind.

For example, every Red Hat Linux system stores its version number in /etc/redhat-release. On my system, it looks like this:

Red Hat Linux release 7.1 (Seawolf)

When applying new RPM files to your system, it is often helpful to know which Red Hat version you're using. On the command line, you can retrieve just that number with:

awk '{print $5}' /etc/redhat-release

What's going on here? By default, awk splits each line read from standard input on whitespace, as is explained below. In effect, it's like you are looking at one row of a spreadsheet. In spreadsheets, columns are usually named with letters. In awk, columns are numbered and you only can see one row (that is, one line of input) at a time. The Red Hat version number is in the fifth column. Similar to the way shells use $ for variable interpolation, the values of columns in awk are retrieved using variables that start with $ and are followed by an integer.

As you can guess, this is a fairly simple demostration of awk, which includes support for regular expressions, branching and looping, and subroutines. For a more complete reference on using awk, see Effective awk Programming or sed & awk Pocket Reference, both published by O'Reilly.

Since there are many flavor of awk, such as nawk and gawk (Section 18.11), this article tries to provide a usable reference for the most common elements of the language. Dialect differences, when they occur, are noted. With the exception of array subscripts, values in

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

awk was introduced as part of Unix's seventh edition and has been part of the standard distribution ever since.

In 1985, the authors of awk extended the language, adding many useful features. Unfortunately, this new version remained inside AT&T for several years. It became a regular part of AT&T's System V as of Release 3.1. It can be found under the name of nawk (for "new awk"); the older version still exists under its original name.

Go to http://examples.oreilly.com/upt3 for more information on: gawk

Unfortunately, nawk is not available on all systems. The good news is that the Free Software Foundation GNU project's version of awk, called gawk, implements all the features of the new awk.

In general, you can assume that what is true for nawk is true for gawk, unless gawk is explicitly called out. Scripts written for nawk are 100 percent compatible with gawk. If you want to use one of the nawk scripts and don't have nawk on your system, simply change the script to invoke gawk instead.

There are a few areas where gawk has introduced gawk-specific features; however, recent versions of nawk support many of these features, suggesting that the remaining differences are really very minor. This shouldn't matter in any case, since we do supply gawk on the disc.

This book doesn't cover any of the awks in detail. The recent awks, especially, have quite a few features in common — so documentation for any one of them should help you learn the others.

In this book, we show scripts for awk as well as scripts that work only on nawk and gawk. But we barely scratch the surface of all the awks' features. The completely revised second edition of O'Reilly & Associates' sed & awk has detailed information on all of them. There's also lots of documentation on the CD-ROM [see http://examples.oreilly.com/upt3]. Take your pick.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are many specialized forms of editing that happen frequently enough that they sometimes want to be saved into a script. Examples of this kind of thing include:

fmt (Section 21.2) and related scripts (Section 21.3) for reformatting jagged lines into neat paragraphs
recomment (Section 21.4), a script for reformatting comment blocks within programs and scripts
behead (Section 21.5), a script for removing the headers from mail and news messages
center (Section 21.8), a script for centering lines of text in a file

In addition, there are a number of programs that provide some useful ways of modifying files but that you don't normally think of as editors:

split (Section 21.9) and csplit (Section 21.10) let you split a big file into smaller pieces.
tr (Section 21.11) lets you substitute one character for another — including non-printing characters that you specify by their octal values.
dd (Section 21.6, Section 21.13) lets you perform various data conversions on a file.
cut (Section 21.14) lets you cut columns or fields out of a file, and paste (Section 21.18) lets you put them back, perhaps in a different order.

This chapter covers all that and more.

— TOR

One of the problems with fold is that it breaks text at an arbitrary column position — even if that position happens to be in the middle of a word. It's a pretty primitive utility, designed to keep long lines from printing off the edge of a line printer page, and not much more.

fmt can do a better job because it thinks in terms of language constructs like paragraphs. fmt wraps lines continuously, rather than just folding the long ones. It assumes that paragraphs end at blank lines.

You can use fmt for things like neatening lines of a mail message or a file that you're editing

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are many specialized forms of editing that happen frequently enough that they sometimes want to be saved into a script. Examples of this kind of thing include:

fmt (Section 21.2) and related scripts (Section 21.3) for reformatting jagged lines into neat paragraphs
recomment (Section 21.4), a script for reformatting comment blocks within programs and scripts
behead (Section 21.5), a script for removing the headers from mail and news messages
center (Section 21.8), a script for centering lines of text in a file

In addition, there are a number of programs that provide some useful ways of modifying files but that you don't normally think of as editors:

split (Section 21.9) and csplit (Section 21.10) let you split a big file into smaller pieces.
tr (Section 21.11) lets you substitute one character for another — including non-printing characters that you specify by their octal values.
dd (Section 21.6, Section 21.13) lets you perform various data conversions on a file.
cut (Section 21.14) lets you cut columns or fields out of a file, and paste (Section 21.18) lets you put them back, perhaps in a different order.

This chapter covers all that and more.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the problems with fold is that it breaks text at an arbitrary column position — even if that position happens to be in the middle of a word. It's a pretty primitive utility, designed to keep long lines from printing off the edge of a line printer page, and not much more.

fmt can do a better job because it thinks in terms of language constructs like paragraphs. fmt wraps lines continuously, rather than just folding the long ones. It assumes that paragraphs end at blank lines.

You can use fmt for things like neatening lines of a mail message or a file that you're editing with vi (Section 17.28). (Emacs has its own built-in line-neatener.) It's also great for shell programming and almost any place you have lines that are too long or too short for your screen.

To make this discussion more concrete, let's imagine that you have the following paragraph:

    Most people take their  Emo Phillips  for granted.  They figure, and not

without some truth, that he is a God-given right and any government that

considers   itself a democracy would naturally provide

its citizens with this

sort of access.  But what too many of this  Gap-wearing,

Real World-watching generation  fail to realize

is that our American

forefathers, under  the  tutelage of Zog, the wizened master sage from

Zeta-Reticuli, had to fight  not only   the godless and   effete British

for our system of  self-determined government, but also  avoid the  terrors

of hynpo-death  from the dark and

unclean Draco-Repitilians.

To prepare this text for printing, you'd like to have all the lines be about 60 characters wide and remove the extra space in the lines. Although you could format this text by hand, GNU fmt can do this for you with the following command line:

% fmt -tuw 60 my_file

The -t option, short for --tagged-paragraph mode, tells fmt to preserve the paragraph's initial indent but align the rest of the lines with the left margin of the second line. The -u option, short for --uniform-spacing, squashes all the inappropriate whitespace in the lines. The final option,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

fmt (Section 21.2) is hard to do without once you've learned about it. Unfortunately, it's not available in some versions of Unix. You can get the GNU version from the CD-ROM [see http://examples.oreilly.com/upt3], but it's also relatively easy to emulate with sed (Section 37.4) and nroff . Using those two utilities also lets you take advantage of the more sophisticated formatting and flexibility that sed and nroff macros can give you. (If you're doing anything really fancy, like tables with tbl, you might need col or colcrt to clean up nroff's output.)

Here's the script:

Go to http://examples.oreilly.com/upt3 for more information on: fmt.sh

#!/bin/sh

sed '1i\

.ll 72\

.na\

.hy 0\

.pl 1' $* | nroff

The reason this is so complicated is that, by default, nroff makes some assumptions you need to change. For example, it assumes an 11-inch page (66 lines) and will add blank lines to a short file (or the end of a long file). The quick-and-dirty workaround to this is to manually put the nroff request .pl 1 (page length 1 line) at the top of the text you want to reformat. nroff also tends to justify lines; you want to turn this off with the .na request. You also want to turn off hyphenation (.hy 0), and you may want to set the line length to 72 instead of nroff's default 65, if only for consistency with the real fmt program. All these nroff requests get inserted before the first line of input by the sed 1i command.

A fancier script would take a -nn line-length option and turn it into a .ll request for nroff, etc.

Another solution to consider is Damian Conway's Text::Autoformat Perl module. It has some very sophisticated heurestics to try to figure out how text should be formatted, including bulleted and numbered lists. In its simplest form, it can be used to read from stdin and write to stdout, just as a standard Unix utility would do. You can invoke this module from the command line like this:

% perl -MText::Autoformat -e 'autoformat' < your_file_here

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Lines in a program's comment block usually start with one or more special characters, like this:

# line 1 of the comment

# line 2 of the comment

# line 3 of the comment

    ...

It can be a hassle to add more text to one of the comment lines in a block, because the line can get too long, which requires you to fold that line onto the next line, which means you have to work around the leading comment character(s).

The fmt (Section 21.2) program neatens lines of a text file. But the standard fmt won't help you "neaten" blocks of comments in a program: it mixes the comment characters from the starts of lines with the words. (If your fmt has the -p option, it handles this problem; there's an example below.) The recomment script is fmt for comment blocks. It's for people who write shell, awk, C, or almost any other kind of program with comment blocks several lines long.

Go to http://examples.oreilly.com/upt3 for more information on: recomment

recomment reads the lines that you feed its standard input. It looks at the first line and figures out what characters you're using to comment the line (see the $cchars variable for a list — typically SPACEs, TABs, #, or *). recomment then strips those comment characters off each line, feeds the remaining block of text to the fmt utility, and uses sed (Section 34.1) to add the comment characters again.

I usually use recomment from inside vi, with filter-through (Section 17.18) commands like:

!}recomment  reformat to the next blank line

5!!recomment reformat this line and the next 4

Normally, recomment lets fmt choose the width of the comment block (72 characters, typically). To get another width, you can do one of the following:

Give the width on the command line, like this:
```
recomment -50
```
Set an environment variable named CBLKWID . Give the maximum width, in characters, for the comment text. For example, in the C shell, use:
```
% setenv CBLKWID 50

                     
```

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you're saving or resending a Usenet article or mail message, you might want to the remove header lines (Subject:, Received:, and so on). This little script will handle standard input, one or many files. It writes to standard output. Here are a few examples:

With saved messages, at a shell prompt:

mail Section 1.21

% behead msg* | mail -s "Did you see these?" fredf

To save an article from a pipe without a header, from a program (here, the old readnews) that can't cut off headers itself:

What now? [ynq] s- | behead > filename

Here's the script, adapted a little from the original by Arthur David Olson:

Go to http://examples.oreilly.com/upt3 for more information on: behead

#! /bin/sh

case $# in

0)  exec sed '1,/^$/d' ;;

*)  for i

    do sed '1,/^$/d' "$i"

    done

    ;;

esac

The script relies on the fact that news articles and mail messages use a blank line to separate the header from the body of the message. As a result, the script simply deletes the text from the beginning up to the first blank line.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Want to strip off some arbitrary number of characters from the front of a file?

Go to http://examples.oreilly.com/upt3 for more information on: dd

dd provides an unexpectedly easy answer. Let's say you wanted to delete the first 100 characters in a file. Here's the command that will do the trick (assuming of course that you give dd a filename with the if= option or data from a pipe):

% dd bs=100 skip=1

Or you could try:

% dd bs=1 skip=100

dd normally reads and writes data in 512-byte blocks; the input block size can be changed with the ibs= option, and the output block size with obs=. Use bs= to set both. skip= sets the number of blocks to skip at the start of the file.

Why would you want to do this? Section 21.9 gives an interesting example of reading text from standard input and writing it to a series of smaller files. Section 21.13 shows even more uses for dd.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Do you have a printer that starts each line too close to the left margin? You might want to indent text to make it look better on the screen or a printed page. Here's a Perl script that does that. It reads from files or standard input and writes to standard output. The default indentation is 5 spaces. For example, to send a copy of a file named graph to the lp printer, indented 12 spaces:

% offset -12 graph | lp

Here's the Perl script that does the job:

Go to http://examples.oreilly.com/upt3 for more information on: offset

#!/usr/local/bin/perl



if ($ARGV[0] =~ /-[0-9]+/) {

    ($indent = $ARGV[0]) =~ s/-//;

    shift @ARGV;

} else {

    $indent = 5;

}



while (<>) {

    print " " x $indent, $_;

}

If there's an indentation amount in the first command-line argument, the dash is stripped and the value stored, then that argument is shifted away. Then a loop steps through the remaining arguments, if any (otherwise standard input is read) and outputs their text preceded by spaces. The script uses the Perl operator "string" x n, which outputs the string (in this case, a single space) n times. The Perl $_ operator contains the current input line.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's an awk script, written by Greg Ubben, that centers text across an 80-character line. If your system understands #! (Section 36.3), this script will be passed directly to awk without a shell. Otherwise, put this into a Bourne shell wrapper (Section 35.19).

Go to http://examples.oreilly.com/upt3 for more information on: center

#!/usr/bin/awk -f

{

    printf "%" int(40+length($0)/2) "s\n", $0

}

For each input line, the script builds a printf command with a width specification just wide enough to center the line (which awk holds in $0). For instance, a line 60 characters wide would give a value of int(40+60/2), which is 70. That makes the following printf command:

printf %70s\n, $0

Because %s prints a string right-justified, that command gives a 10-character indentation (70 minus 60) on an 80-character line. The right end of the line is also 10 characters (80 minus 70) from the right edge of the screen.

In vi , you can use a filter-through (Section 17.18) command to center lines while you're editing. Or just use center from the command line. For example:

% center afile > afile.centered

% sort party_list | center | lp

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most versions of Unix come with a program called split whose purpose is to split large files into smaller files for tasks such as editing them in an editor that cannot handle large files, or mailing them if they are so big that some mailers will refuse to deal with them. For example, let's say you have a really big text file that you want to mail to someone:

% ls -l bigfile

-r--r--r--  1 jik        139070 Oct 15 21:02 bigfile

Running split on that file will (by default, with most versions of split) break it up into pieces that are each no more than 1000 lines long:

wc Section 16.6

% split bigfile

% ls -l

total 283

-r--r--r--  1 jik        139070 Oct 15 21:02 bigfile

-rw-rw-r--  1 jik         46444 Oct 15 21:04 xaa

-rw-rw-r--  1 jik         51619 Oct 15 21:04 xab

-rw-rw-r--  1 jik         41007 Oct 15 21:04 xac

% wc -l x*

    1000 xaa

    1000 xab

     932 xac

    2932 total

Note the default naming scheme, which is to append "aa", "ab", "ac", etc., to the letter "x" for each subsequent filename. It is possible to modify the default behavior. For example, you can make split create files that are 1500 lines long instead of 1000:

% rm x??

% split -1500 bigfile

% ls -l

total 288

-r--r--r--  1 jik        139070 Oct 15 21:02 bigfile

-rw-rw-r--  1 jik         74016 Oct 15 21:06 xaa

-rw-rw-r--  1 jik         65054 Oct 15 21:06 xab

You can also get it to use a name prefix other than "x":

% rm x??

% split -1500 bigfile bigfile.split.

% ls -l

total 288

-r--r--r--  1 jik        139070 Oct 15 21:02 bigfile

-rw-rw-r--  1 jik         74016 Oct 15 21:07 bigfile.split.aa

-rw-rw-r--  1 jik         65054 Oct 15 21:07 bigfile.split.ab

Although the simple behavior described above tends to be relatively universal, there are differences in the functionality of split on different Unix systems. There are four basic variants of split as shipped with various implementations of Unix:

A split that understands only how to deal with splitting text files into chunks of n lines or less each.
A split, usually called bsplit

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: csplit

Like split (Section 21.9), csplit lets you break a file into smaller pieces, but csplit (context split) also allows the file to be broken into different-sized pieces, according to context. With csplit, you give the locations (line numbers or search patterns) at which to break each section. csplit comes with System V, but there are also free versions available.

Let's look at search patterns first. Suppose you have an outline consisting of three main sections that start on lines with the Roman numerals I., II., and III.. You could create a separate file for each section by typing:

% csplit outline /I./ /II./ /III./

28      number of characters in each file

415                   .

372                   .

554                   .

% ls

outline

xx00     outline title, etc.

xx01     Section I

xx02     Section II

xx03     Section III

This command creates four new files (outline remains intact). csplit displays the character counts for each file. Note that the first file (xx00) contains any text up to but not including the first pattern, and xx01 contains the first section, as you'd expect. This is why the naming scheme begins with 00. (If outline had begun immediately with a I., xx01 would still contain Section I, but in this case xx00 would be empty.)

If you don't want to save the text that occurs before a specified pattern, use a percent sign as the pattern delimiter:

% csplit outline %I.% /II./ /III./

415

372

554

% ls

outline

xx00         Section I

xx01         Section II

xx02         Section III

The preliminary text file has been suppressed, and the created files now begin where the actual outline starts (the file numbering is off, however).

Let's make some further refinements. We'll use the -s option to suppress the display of the character counts, and we'll use the -f option to specify a file prefix other than the conventional

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The tr command is a character translation filter, reading standard input (Section 43.1) and either deleting specific characters or substituting one character for another.

The most common use of tr is to change each character in one string to the corresponding character in a second string. (A string of consecutive ASCII characters can be represented as a hyphen-separated range.)

For example, the command:

< Section 43.1

$ tr 'A-Z' 'a-z' <

                file        

               Berkeley version

will convert all uppercase characters in file to the equivalent lowercase characters. The result is printed on standard output.

In fact, a frequent trick I use tr for is to convert filenames from all uppercase to all lowercase. This comes up when you're dealing with files from MS-DOS or VMS that you are copying on to a Unix filesystem. To change all the files in the current directory to uppercase, try this from a Bash or Bourne shell prompt:

$ for i in `ls`; do mv $i `echo $i | tr [A-Z] [a-z]`; done

Of course, you need to be careful that there are no files that have the same name regardless of case. The GNU mv can be passed the -i flag that will make the program prompt you before overwriting an existing file. If you want to uppercase filenames, simply flip the arguments to tr. You can even apply this to an entire branch of a file system by sticking this in a find command. First, create a small shell script that can downcase a file and call it downcase:

#!/bin/sh

mv $1 `echo $1 | tr [A-Z] [a-z]`

Now you can really do some damage with find:

$ find /directory/to/be/affected -exec 'downcase' '{}' ';'

Obviously, running this programming on random directories as root is not recomended, unless you're looking to test your backup system.

Go to http://examples.oreilly.com/upt3 for more information on: tr

In the System V version of tr, square brackets must surround any range of characters. That is, you have to say

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Email transport systems were originally designed to transmit characters with a seven-bit encoding — like ASCII. This meant they could send messages with plain English text but not "binary" text, such as program files or graphics (or non-English text!), that used all of an eight-bit byte. Usenet (Section 1.21), the newsgroup system, was transmitted like email and had its same seven-bit limitations. The solution — which is still used today — is to encode eight-bit text into characters that use only the seven low bits.

The first popular solution on Unix-type systems was uuencoding . That method is mostly obsolete now (though you'll still find it used sometimes); it's been replaced by MIME encoding. The next two sections cover both of those — though we recommend avoiding uuencode like the plague.

The uuencode utility encodes eight-bit data into a seven-bit representation for sending via email or on Usenet. The recipient can use uudecode to restore the original data. Unfortunately, there are several different and incompatible versions of these two utilities. Also, uuencoded data doesn't travel well through all mail gateways — partly because uuencoding is sensitive to changes in whitespace (space and TAB) characters, and some gateways munge (change or corrupt) whitespace. So if you're encoding text for transmission, use MIME instead of uuencode whenever you can.

To create an ASCII version of a binary file, use the uuencode utility. For instance, a compressed file (Section 15.6) is definitely eight-bit; it needs encoding.

A uuencoded file (there's an example later in this article) starts with a begin line that gives the file's name; this name comes from the first argument you give the uuencode utility as it encodes a file. To make uuencode read a file directly, give the filename as the second argument. uuencode writes the encoded file to its standard output. For example, to encode the file emacs.tar.gz

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Besides the other uses of dd (Section 21.6) we've covered, you also can use this versatile utility to convert:

fixed length to variable-length records (conv=unblock), and the reverse (conv=block)
uppercase to lowercase (conv=lcase), and the reverse (conv=ucase)
the byte order of every pair of bytes (conv=swab)
ASCII to EBCDIC and the reverse (conv=ebcdic, conv=ibm). If you're converting old IBM tapes, you'll need to know the tape's blocking factor. And if the tape has multiple files on it, you'll have to use the tape device name that allows "no rewind on close" (Section 38.5) to read past the first file.

The cbs= option must be used to specify a conversion buffer size when using block and unblock and when converting between ASCII and EBCDIC. The specified number of characters are put into the conversion buffer. For ascii and unblock conversion, trailing blanks are trimmed and a newline is added to each buffer before it is output. For ebcdic, ibm, and block, the input is padded with blanks up to the specified conversion buffer size.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A nifty command called cut lets you select a list of columns or fields from one or more files.

You must specify either the -c option to cut by column or -f to cut by fields. (Fields are separated by tabs unless you specify a different field separator with -d. Use quotes (Section 27.12) if you want a space or other special character as the delimiter.)

In some versions of cut, the column(s) or field(s) to cut must follow the option immediately, without any space. Use a comma between separate values and a hyphen to specify a range (e.g., 1-10,15 or 20,23 or 50-).

The order of the columns and fields is ignored; the characters in each line are always output from first to last, in the order they're read from the input. For example, cut -f1,2,4 produces exactly the same output as cut -f4,2,1. If this isn't what you want, try perl (Section 41.1) or awk (Section 20.10), which let you output fields in any order.

cut is incredibly handy. Here are some examples:

Find out who is logged in, but list only login names:
who Section 2.8
```
% who | cut -d" " -f1

                  
```

Extract usernames and real names from /etc/passwd (Section 22.3):

% cut -d: -f1,5 /etc/passwd

Cut characters in the fourth column of file, and paste them back as the first column in the same file:

% cut -c4 file | paste - file

Section 21.18 covers the cut counterpart, paste.

As was mentioned, you can use awk or perl to extract columns of text. Given the above task to extract the fifth and first fields fields of /etc/passwd, you can use awk:

% awk -F: '{print $5, "=>", $1}' /etc/passwd

An often forgotten command-line option for perl is -a, which puts perl in awk compatibility mode. In other words, you can get the same field-splitting behavior right from the command line:

% perl -F: -lane 'print $F[4], "=>", "$F[0]"' /etc/passwd

In the line above,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The pr command (Section 45.6) is famous for printing a file neatly on a page — with margins at top and bottom, filename, date, and page numbers. It can also print text in columns: one file per column or many columns for each file.

The -t option takes away the heading and margins at the top and bottom of each page. That's useful when "pasting" data into columns with no interruptions.

The -m option reads all files on the command line simultaneously and prints each in its own column, like this:

% file1 file2 file3



The lines              The lines              The lines

of file1               of file2               of file3

are here               are here               are here

  ...                    ...                    ...

pr may use TAB characters between columns. If that would be bad, you can pipe pr's output through expand. Many versions of pr have a -s X option that sets the column separator to the single character X.

By default, pr -m doesn't put filenames in the heading. If you want that, use the -h option to make your own heading. Or maybe you'd like to make a more descriptive heading. Here's an example using process substitution to compare a directory with its RCS (Section 39.5) subdirectory:

% pr -m -h "working directory compared to RCS directory" <(ls) <(ls RCS)



2000-11-22 23:57  working directory compared to RCS directory  Page    1



0001.sgm                            0001.sgm,v

0002.sgm                            0002.sgm,v

0007.sgm                            0007.sgm,v

0008.sgm                            0008.sgm,v

             ...

(The heading comes from the GNU version of pr. Later examples in this article use a different version with a different heading format.)

An option that's a number will print a file in that number of columns. For instance, the -3 option prints a file in three columns. The file is read, line by line, until the first column is full (by default, that takes 56 lines). Next, the second column is filled. Then, the third column is filled. If there's more of the file, the first column of page 2 is filled — and the cycle repeats:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: column

Another column-making program, besides cols and pr (Section 21.15), is the creatively named utility column. It tries to determine the terminal width, which you can override with the -c option (-c 132, for example, gives 132 columns: handy for printing on wide line-printer paper.) The -x option fills columns before rows — similar to pr with its - n option and cols -d.

What makes column different from the others is its -t option. This reads input data that's already in columns and rebalances the columns into a table with variable-width columns. Say what? This is easiest to see with an example, and the column(1) manual page has a good one.

If you'd like to add column headings to ls -l output, it can be a pain to try to make headings that each take the same number of characters as the data below them. For instance, the first field on each line, the permissions, takes 10 characters, but if you want to use the heading "PERM", which takes only 4 characters, you need to balance it by adding 6 spaces after. Using column -t, you can balance these automatically. Here's an example. The first command is plain ls -l. In the second and third examples, I use sed 1d (Section 34.1) to delete the total n line from ls, and subshells (Section 24.4) to make both commands use the same standard output; this is important only in the third command, where I pipe the combined stdout to column for balancing:

; Section 28.16, > Section 28.12

$ ls -lo

total 1644

-r--r--r--    1 jpeek     1559177 Sep 19  1999 ORA_tifs.tgz

-rw-rw-r--    1 jpeek        4106 Oct 21  1999 UPT_Russian.jpg

-rw-rw-r--    1 jpeek      101944 Nov 19 09:30 london_dusk-livesights.xwd.gz

dr-xr-xr-x    2 jpeek        4096 Dec 12  1999 me

$ (echo "PERM      LINKS OWNER        SIZE MON DY TM/YR NAME"; \

> ls -lo | sed 1d)

PERM      LINKS OWNER        SIZE MON DY TM/YR NAME

-r--r--r--    1 jpeek     1559177 Sep 19  1999 ORA_tifs.tgz

-rw-rw-r--    1 jpeek        4106 Oct 21  1999 UPT_Russian.jpg

-rw-rw-r--    1 jpeek      101944 Nov 19 09:30 london_dusk-livesights.xwd.gz

dr-xr-xr-x    2 jpeek        4096 Dec 12  1999 me



$

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As we were writing this book, I decided to make a list of all the articles and the numbers of lines and characters in each, then combine that with the description, a status code, and the article's title. After a few minutes with wc -l -c (Section 16.6), cut (Section 21.14), sort (Section 22.1), and join (Section 21.19), I had a file that looked like this:

% cat messfile

2850 2095 51441 ~BB A sed tutorial

3120 868 21259 +BB mail - lots of basics

6480 732 31034 + How to find sources - JIK's periodic posting

    ...900 lines...

5630 14 453 +JP Running Commands on Directory Stacks

1600 12 420 !JP With find, Don't Forget -print

0495 9 399 + Make 'xargs -i' use more than one filename

Yuck. It was tough to read: the columns needed to be straightened. The column (Section 21.16) command could do it automatically, but I wanted more control over the alignment of each column. A little awk (Section 20.10) script turned the mess into this:

% cat cleanfile

2850 2095  51441 ~BB  A sed tutorial

3120  868  21259 +BB  mail - lots of basics

6480  732  31034 +    How to find sources - JIK's periodic posting

    ...900 lines...

5630   14    453 +JP  Running Commands on Directory Stacks

1600   12    420 !JP  With find, Don't Forget -print

0495    9    399 +    Make 'xargs -i' use more than one filename

Here's the simple script I used and the command I typed to run it:

% cat neatcols

{

printf "%4s %4s %6s %-4s %s\n", \

     $1, $2, $3, $4, substr($0, index($0,$5))

}

% awk -f neatcols messfile > cleanfile

You can adapt that script for whatever kinds of columns you need to clean up. In case you don't know awk, here's a quick summary:

The first line of the printf, between double quotes ("), specifies the field widths and alignments. For example, the first column should be right-aligned in 4 characters (%4s). The fourth column should be 4 characters wide left-adjusted (%-4s). The fifth column is big enough to just fit (%s). I used string (%s) instead of decimal (%d) so awk wouldn't strip off the leading zeros in the columns.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: cut+paste

Do you ever wish you could paste two (or even three) files side by side? You can, if you have the paste program (or the public-domain implementation on the disc).

For example, to create a three-column file from files x, y, and z:

$ paste x y z > file

To make paste read standard input, use the - option, and repeat - for every column you want. For example, to make an old ls (which lists files in a single column) list files in four columns:

$ ls | paste - - - -

The "standard input" option is also handy when used with cut (Section 21.14). You can cut data from one position on a line and paste it back on another.

The separate data streams being merged are separated by default with a tab, but you can change this with the -d option. Unlike the -d option to cut, you need not specify a single character; instead, you can specify a list of characters, which will be used in a circular fashion.

The characters in the list can be any regular character or the following escape sequences:

\n: newline
\t: tab
\\: backslash
\0: empty string

Use quoting (Section 27.12), if necessary, to protect characters from the shell.

There's also a -s option that lets you merge subsequent lines from one file. For example, to merge each pair of lines onto a single line:

$ paste -s -d"\t\n" list

Let's finish with one nice place to use process substitution, if your shell has it. You can use cut to grab certain columns from certain files, then use process substitution to make "files" that paste will read. Output those "files" into columns in any order you want. For example, to paste column 1 from file1 in the first output column, and column 3 from file2 in the second output column:

paste <(cut -f1 file1) <(cut -f3 file2)

If none of the shells on your system have process substitution, you can always use a bunch of temporary files, one file per column.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you've worked with databases, you'll probably know what to do with the Unix join command; see your online manual page. If you don't have a database (as far as you know!), you still probably have a use for join: combining or "joining" two column-format files. join searches certain columns in the files; when it finds columns that match one another, it "glues the lines together" at that column. This is easiest to show with an example.

I needed to summarize the information in thousands of email messages under the MH mail system. MH made that easy: it has one command (scan) that gave me almost all the information I wanted about each message and also let me specify the format I needed. But I also had to use wc -l (Section 16.6) to count the number of lines in each message. I ended up with two files, one with scan output and the other with wc output. One field in both lines was the message number; I used sort (Section 22.1) to sort the files on that field. I used awk '{print $1 "," $2}' to massage wc output into comma-separated fields. Then I used join to "glue" the two lines together on the message-number field. (Next I fed the file to a PC running dBASE, but that's another story.)

Here's the file that I told scan to output. The columns (message number, email address, comment, name, and date sent) are separated with commas (,):

0001,andrewe@isc.uci.edu,,Andy Ernbaum,19901219

0002,bc3170x@cornell.bitnet,,Zoe Doan,19910104

0003,zcode!postman@uunet.uu.net,,Head Honcho,19910105

   ...

Here's the file from wc and awk with the message number and number of lines:

The following join command then joined the two files at their first columns (-t, tells join that the fields are comma-separated):

% join -t, scanfile wcfile

The output file looked like this:

0001,andrewe@isc.uci.edu,,Andy Ernbaum,19901219,11

0002,bc3170x@cornell.bitnet,,Zoe Doan,19910104,5

0003,zcode!postman@uunet.uu.net,,Head Honcho,19910105,187

   ...

Go to http://examples.oreilly.com/upt3 for more information on: join

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: uniq

uniq reads a file and compares adjacent lines (which means you'll usually want to sort the file first to be sure identical lines appear next to each other). Here's what uniq can do as it watches the input lines stream by:

With the -u option, the output gets only the lines that occur just once (and weren't repeated).
The -d option does the opposite: the output gets a single copy of each line that was repeated (no matter how many times it was repeated).
(The GNU version also has a -D option. It's like -d except that all duplicate lines are output.)
The default output (with no options) is the union of -u and -d: only the first occurrence of a line is written to the output file; any adjacent copies of a line (second, third, etc.) are ignored.
The output with -c is like the default, but each line is preceded by a count of how many times it occurred.

Be warned:

% uniq file1 file2

will not print the unique lines from both file1 and file2 to standard output. It will replace the contents of file2 with the unique lines from file1!

Three more options control how comparisons are done:

- n ignores the first n fields of a line and all whitespace before each. A field is defined as a string of nonwhitespace characters (separated from its neighbors by whitespace).
+ n ignores the first n characters. Fields are skipped before characters.
-w n in the GNU version compares no more than n characters in each line.
GNU uniq also has -i to make comparisons case-insensitive. (Upper- and lowercase letters compare equal.)

uniq is often used as a filter. See also comm (Section 11.8), sort (Section 22.1), and especially sort -u (Section 22.6).

So what can you do with all of this?

To send only one copy of each line from

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Every now and then you come across something and say, "Gee, that might come in handy someday, but I have no idea for what." This might happen to you when you're browsing at a flea market or garage sale; if you're like us, it might happen when you're browsing through public domain software.

Go to http://examples.oreilly.com/upt3 for more information on: rot

Which brings us to the rot program. rot basically just rotates text columns and rows. For example, the first column below shows an input file. The other three columns show the same file fed through rot once, twice, and three times:

$ cat file	$ rot file	$ rot file \| rot	$ rot file \| rot \| rot
abcde	54321	5	e
1	a	4	d
2	b	3	c
3	c	2	b
4	d	1	a
5

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sorting a file under Unix is easy, right? Of course it is, if all you want to do is sort a list of single words, or sort lines starting with the first character in the line. But if you want to do more than that, there's a lot more to the sort command than typing sort filename:

Section 22.2 describes how to select individual fields from a line for sort to operate on.
Section 22.3 describes how to change the field delimiter from whitespace to some other character.
Section 22.4 describes the kinds of problems that you can encounter if fields are delimited by whitespace.
Section 22.5 clarifies the distinctions between alphabetic and numeric sorting.
Section 22.6 gives miscellaneous hints about useful sort options.

But learning the mechanics of sort isn't the end of the story. Like most of the other things you'll find in the Unix toolbox, sort is even more powerful when it's used with other programs. For example, you can:

Sort paragraphs or other multiline entries.
Sort lines by how long they are (Section 22.7).
Sort a list of names by last name, whether or not there's a middle name as well (Section 22.8).

— TOR

Unless you tell it otherwise, sort divides each line into fields at whitespace (blanks or tabs), and sorts the lines by field, from left to right.

That is, it sorts on the basis of field 0 (leftmost), but when the leftmost fields are the same, it sorts on the basis of field 1, and so on. This is hard to put into words, but it's really just common sense. Suppose your office inventory manager created a file like this:

supplies     pencils  148

furniture    chairs   40

kitchen      knives   22

kitchen      forks    20

supplies     pens     236

furniture    couches  10

furniture    tables   7

supplies     paper    29

You'd want all the supplies sorted into categories, and within each category, you'd want them sorted alphabetically:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sorting a file under Unix is easy, right? Of course it is, if all you want to do is sort a list of single words, or sort lines starting with the first character in the line. But if you want to do more than that, there's a lot more to the sort command than typing sort filename:

Section 22.2 describes how to select individual fields from a line for sort to operate on.
Section 22.3 describes how to change the field delimiter from whitespace to some other character.
Section 22.4 describes the kinds of problems that you can encounter if fields are delimited by whitespace.
Section 22.5 clarifies the distinctions between alphabetic and numeric sorting.
Section 22.6 gives miscellaneous hints about useful sort options.

But learning the mechanics of sort isn't the end of the story. Like most of the other things you'll find in the Unix toolbox, sort is even more powerful when it's used with other programs. For example, you can:

Sort paragraphs or other multiline entries.
Sort lines by how long they are (Section 22.7).
Sort a list of names by last name, whether or not there's a middle name as well (Section 22.8).

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unless you tell it otherwise, sort divides each line into fields at whitespace (blanks or tabs), and sorts the lines by field, from left to right.

That is, it sorts on the basis of field 0 (leftmost), but when the leftmost fields are the same, it sorts on the basis of field 1, and so on. This is hard to put into words, but it's really just common sense. Suppose your office inventory manager created a file like this:

supplies     pencils  148

furniture    chairs   40

kitchen      knives   22

kitchen      forks    20

supplies     pens     236

furniture    couches  10

furniture    tables   7

supplies     paper    29

You'd want all the supplies sorted into categories, and within each category, you'd want them sorted alphabetically:

% sort supplies

furniture    chairs   40

furniture    couches  10

furniture    tables   7

kitchen      forks    20

kitchen      knives   22

supplies     paper    29

supplies     pencils  148

supplies     pens     236

Of course, you don't always want to sort from left to right. The command-line option +n tells sort to start sorting on field n; -n tells sort to stop sorting on field n. Remember (again) that sort counts fields from left to right, starting with 0. Here's an example. We want to sort a list of telephone numbers of authors, presidents, and blues singers:

Robert M Johnson      344-0909

Lyndon B Johnson      933-1423

Samuel H Johnson      754-2542

Michael K Loukides    112-2535

Jerry O Peek          267-2345

Timothy F O'Reilly    443-2434

According to standard "telephone book rules," we want these names sorted by last name, first name, and middle initial. We don't want the phone number to play a part in the sorting. So we want to start sorting on field 2, stop sorting on field 3, continue sorting on field 0, sort on field 1, and (just to make sure) stop sorting on field 2 (the last name). We can code this as follows:

% sort +2 -3 +0 -2 phonelist

Lyndon B Johnson      933-1423

Robert M Johnson      344-0909

Samuel H Johnson      754-2542

Michael K Loukides    112-2535

Timothy F O'Reilly    443-2434

Jerry O Peek          267-2345

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 22.2 explained how sort separates a line of input into two or more fields using whitespace (spaces or tabs) as field delimiters. The -t option lets you change the field delimiter to some other character.

For example, if you wanted to sort the login names on your system by the login shell they use, you could issue a command like this:

/etc..wd Section 1.7

% sort -t: +6 /etc/passwd

root:SndEKOs9H7YLm:0:1:Operator:/:/bin/bash

sys:*:2:2::/:/bin/bash

jim:LjKwcUt8l6kZK:2391:1004:Jim O'Callahan:/u/jim:/bin/bash

   ...

bart:2DPD8rCOKBbUu:2665:1004:Bart Buus:/u/bart:/bin/tcsh

tap:xY7oeuJ8WxyGO:2943:1004:Tap Bronman:/u/tap:/bin/tcsh

The option -t: tells sort to use a colon as a field separator — so, in this example, field 0 is the login name, field 1 is the encoded password, field 2 is the user ID number, field 3 is the group ID number, and so on. By this numbering, the login shell is in the sixth field.

Remember that sort numbers fields starting with zero — this will save you lots of grief. Two consecutive colons indicate a "null" field that still must be counted.

—ML and TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One would hope that a simple task like sorting would be relatively unambiguous. Unfortunately, it isn't. The behavior of sort can be very puzzling. I'll try to straighten out some of the confusion — at the same time, I'll be leaving myself open to abuse by the real sort experts. I hope you appreciate this! Seriously, though: if you know of any other wrinkles to the story, please let us know and we'll add them in the next edition.

The trouble with sort is figuring out where one field ends and another begins. It's simplest if you can specify an explicit field delimiter (Section 22.3). This makes it easy to tell where fields end and begin. But by default, sort uses whitespace characters (tabs and spaces) to separate fields, and the rules for interpreting whitespace field delimiters are unfortunately complicated. As I see them, they are:

The first whitespace character you encounter is a "field delimiter"; it marks the end of the old field and the beginning of the next field.
Any whitespace character following a field delimiter is part of the new field. That is, if you have two or more whitespace characters in a row, the first one is used as a field delimiter and isn't sorted. The remainder are sorted, as part of the next field.
Every field has at least one nonwhitespace character, unless you're at the end of the line. (That is, null fields only occur when you've reached the end of a line.)
All whitespace is not equal. Sorting is done according to the ASCII collating sequence. Therefore, TABs are sorted before spaces.

Here is a silly but instructive example that demonstrates most of the hard cases. We'll sort the file sortme, which is:

        apple   Fruit shipment

20      beta    beta test sites

 5              Something or other

All is not as it seems -- cat -t -v (Section 12.5, Section 12.4) shows that the file really looks like this:

^Iapple^IFruit shipment

20^Ibeta^Ibeta test sites

 5^I^ISomething or other

^I indicates a tab character. Before showing you what sort does with this file, let's break it into fields, being very careful to apply the rules above. In the table, we use quotes to show exactly where each field begins and ends:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

sort performs two fundamentally different kinds of sorting operations: alphabetic sorts and numeric sorts. An alphabetic sort is performed according to the traditional "dictionary order," using the ASCII collating sequence. Uppercase letters come before lowercase letters (unless you specify the -f option, which "folds" uppercase and lowercase together), with numerals and punctuation interspersed. The -l (lowercase L) option sorts by the current locale instead of the default US/ASCII order.

This is all fairly trivial and common sense. However, it's worth belaboring the difference, because it's a frequent source of bugs in shell scripts. Say you sort the numbers 1 through 12. A numeric sort gives you these numbers "in order," just like you'd expect. An alphabetic sort gives you:

Of course, this is how you'd sort the numbers if you applied dictionary rules to the list. Numeric sorts can handle decimal numbers (for example, numbers like 123.44565778); they can't handle floating-point numbers (for example, 1.2344565778E+02). The GNU sort does provide the -g flag for sorting numbers in scientific notation. Unfortunately, it is significantly slower than plain old decimal sorting.

What happens if you include alphabetic characters in a numeric sort? Although the results are predictable, I would prefer to say that they're "undefined." Including alphabetic characters in a numeric sort is a mistake, and there's no guarantee that different versions of sort will handle them the same way. As far as I know, there is no provision for sorting hexadecimal numbers.

One final note: your version of numeric sort may treat initial blanks as significant, sorting numbers with additional spaces before them ahead of numbers without the additional spaces. This is an incredibly stupid misfeature. There is a workaround: use the -b (ignore leading blanks) and always specify a sort field. That is, sort -nb +0 will do what you expect; sort -n won't.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here is a grab bag of useful, if not exactly interesting, sort features. The utility will actually do quite a bit, if you let it.

sort -u sorts the file and eliminates duplicate lines. It's more powerful than uniq (Section 21.20) because:

It sorts the file for you; uniq assumes that the file is already sorted and won't do you any good if it isn't.
It is much more flexible. sort -u considers lines "unique" if the sort fields (Section 22.2) you've selected match. So the lines don't even have to be (strictly speaking) unique; differences outside of the sort fields are ignored.

In return, there are a few things that uniq does that sort won't do — such as print only those lines that aren't repeated, or count the number of times each line is repeated. But on the whole, I find sort -u more useful.

Here's one idea for using sort -u. When I was writing a manual, I often needed to make tables of error messages. The easiest way to do this was to grep the source code for printf statements, write some Emacs (Section 19.1) macros to eliminate junk that I didn't care about, use sort -u to put the messages in order and get rid of duplicates, and write some more Emacs macros to format the error messages into a table. All I had to do then was write the descriptions.

One important option (that I've mentioned a number of times) is -b; this tells sort to ignore extra whitespace at the beginning of each field. This is absolutely essential; otherwise, your sorts will have rather strange results. In my opinion, -b should be the default. But they didn't ask me.

Another thing to remember about -b: it works only if you explicitly specify which fields you want to sort. By itself, sort -b is the same as sort: whitespace characters are counted. I call this a bug, don't you?

If you don't care about the difference between uppercase and lowercase letters, invoke sort with the -f (case-fold) option. This folds lowercase letters into uppercase. In other words, it treats all letters as uppercase.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A nice little script to sort lines from shortest to longest can be handy when you're writing and want to find your big words:

deroff Section 16.9, uniq Section 21.20

% deroff -w report | uniq -d | lensort

a

an

  ...

deoxyribonucleic

Once I used it to sort a list of pathnames:

find Section 9.1

% find adir -type f -print | lensort

adir/.x

adir/.temp

   ...

adir/subdir/part1/somefile

adir/subdir/part1/a_test_case

The script uses awk ( Section 20.10) to print each line's length, followed by the original line. Next, sort sorts the lengths numerically (Section 22.5). Then sed (Section 34.1) strips off the lengths and the spaces and prints the lines:

Go to http://examples.oreilly.com/upt3 for more information on: lensort

#! /bin/sh

awk 'BEGIN { FS=RS }

{ print length, $0 }' $* |

# Sort the lines numerically

sort +0n -1 |

# Remove the length and the space and print each line

sed 's/^[0-9][0-9]* //'

(Some awks require a semicolon after the first curly bracket — that is, { FS=RS };.)

Of course, you can also tackle this problem with Perl:

$ perl -lne '$l{$_}=length;END{for(sort{$l{$a}<=>$l{$b}}keys %l){print}}' \

                filename

This one-line wonder has the side effect of eliminating duplicate lines. If this seems a bit terse, that's because it's meant to be "write-only," that is, it is a bit of shell magic that you'd use to accomplish a short-term task. If you foresee needing this same procedure in the future, it's better to capture the magic in script. Scripts also tend to be easier to understand, debug, and expand. The following script does the same thing as the one-liner but a bit more clearly:

#!/usr/bin/perl



my %lines;

while(my $curr_line = <STDIN>){

  chomp $curr_line;

  $lines{$curr_line} = length $curr_line;

}



for my $line (sort{ $lines{$a} <=> $lines{$b} } keys %lines){

  print $line, "\n";

}

This script reads in a line from standard input, removes the newline character and creates an associative array that maps whole line to its length in characters. After processing the whole file, the keys of the associative array is sorted in ascending numerical order by each key's value. It is then a simple matter to print the key, which is the line itself. More Perl tricks can be found in Chapter 11.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It's hard to sort any old list of peoples' names because some people have one-word first and last names like Joe Smith, but other people have multi-part names like Mary Jo Appleton. This program sorts on the last word in each name. That won't take care of the way that names are used everywhere in the world, but it might give you some ideas.

Go to http://examples.oreilly.com/upt3 for more information on: namesort

The script reads from files or its standard input; it writes to standard output.

#! /bin/sh

# Print last field (last name), a TAB, then whole name:

awk '{print $NF "\t" $0}' $* |

# sort (by last name: the temporary first field)

sort |

# strip off first field and print the names:

cut -f2-

If you want more control over the sorting or you're interested in pulling apart names in general, there's a Perl module you might want to look at called Lingua::EN::NameParse . Below is a Perl script that also sorts a list of names by surname.

#!/usr/bin/perl



use Lingua::EN::NameParse;



my $Name_Obj = Lingua::EN::NameParse->new(auto_clean  => 1);

my @names = <STDIN>;

for my $line (sort by_lastname @names){

  chomp($line);

  print $line, "\n";

}



sub by_lastname {

  my @names;

  for my $name ($a, $b) {

    chomp($name);

    if( my $err = $Name_Obj->parse($name) ){

      warn "WARN: Unparsable name ($name): $err";

    }

    my %tmp = $Name_Obj->components;

    push @names, \%tmp;

  }

  return lc $names[0]->{surname_1} cmp lc $names[1]->{surname_1};

}

The script starts by bringing in the Lingua::EN::NameParse library. Then, all lines from standard input are read in and stored in an array. Perl's sort function is particularly flexible in that it can use a user-defined subroutine to determine the desired collating sequence. Here, the subroutine by_lastname receives the next two items of the list to be sorted in the "magical" global variables $a and $b. These names are then parsed by the global Lingua::EN::NameParse object, and the name components are stored in the array @names. It's then a simple matter to alphabetically compare the lowercased surnames and return that value to

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As has been said many times in this book, Unix is a mutliprocessing system. Unlike some historic systems such as MS-DOS, all flavors of Unix run more than one process at a time. In fact, when Unix boots, the first program executed is called init , which is the parent of all future processes. init immediately creates a new process in which other programs can run, such as getty and the various rc setup scripts. At some point when a user logs into the system, the getty program creates a new shell for that session. Even when the system is in single-user mode, Unix is still capable of running multiple processes. Multiprocessing is prevasive in Unix.

But multiprocessing isn't just for system daemons. It's also there to make your interactive shell session just a little bit more productive. Often, you will need to execute a program that takes a long time to run. For instance, you might be downloading a file with FTP or Lynx. It is possible to have that task put into the background so that you may execute new commands while the previous ones are running to completion. Just as you may have several piles of work on your desk, you often need to set aside one stack to work on another. A process is said to be in the foreground when it is receiving your keyboard input and is writing to your screen. Using the desk analogy, the foreground process is that pile of work currently in front of you. Only one process can be in the foreground at a time. Putting a process in the background is like putting the current stack of work in front of you on the floor. And if your desk is anything like mine, you can soon find your desk surrounded by piles of work. Unlike the real world, Unix is able to continue working on completing processes in the background. The management and manipulation of foreground and background processes is called job control. By understanding job control, you can begin to take better advantage of your Unix system.

One cautionary note on job control: there's no such thing as a free lunch. In other words, while Unix blithely lets you put all the processes you want into the background, they all share the same CPU, RAM, and hard drive resources. If one process dominates one of these resources, the other processes won't get done any faster than they would have had you run them one after the other to completion. So if you've got a process that's CPU-intensive (such as a photomosiac program), there's little point in trying to run more processes on that machine.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As has been said many times in this book, Unix is a mutliprocessing system. Unlike some historic systems such as MS-DOS, all flavors of Unix run more than one process at a time. In fact, when Unix boots, the first program executed is called init , which is the parent of all future processes. init immediately creates a new process in which other programs can run, such as getty and the various rc setup scripts. At some point when a user logs into the system, the getty program creates a new shell for that session. Even when the system is in single-user mode, Unix is still capable of running multiple processes. Multiprocessing is prevasive in Unix.

But multiprocessing isn't just for system daemons. It's also there to make your interactive shell session just a little bit more productive. Often, you will need to execute a program that takes a long time to run. For instance, you might be downloading a file with FTP or Lynx. It is possible to have that task put into the background so that you may execute new commands while the previous ones are running to completion. Just as you may have several piles of work on your desk, you often need to set aside one stack to work on another. A process is said to be in the foreground when it is receiving your keyboard input and is writing to your screen. Using the desk analogy, the foreground process is that pile of work currently in front of you. Only one process can be in the foreground at a time. Putting a process in the background is like putting the current stack of work in front of you on the floor. And if your desk is anything like mine, you can soon find your desk surrounded by piles of work. Unlike the real world, Unix is able to continue working on completing processes in the background. The management and manipulation of foreground and background processes is called job control. By understanding job control, you can begin to take better advantage of your Unix system.

One cautionary note on job control: there's no such thing as a free lunch. In other words, while Unix blithely lets you put all the processes you want into the background, they all share the same CPU, RAM, and hard drive resources. If one process dominates one of these resources, the other processes won't get done any faster than they would have had you run them one after the other to completion. So if you've got a process that's CPU-intensive (such as a photomosiac program), there's little point in trying to run more processes on that machine.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're coming from a Windows or MacOS desktop, Unix job control may seem a little strange at first, but both of those operating systems support a form of job control too. The Windows' taskbar shows the foreground application as a depressed icon. In the classic Mac interface, the current application's icon is present in the upper righthand corner. Such displays aren't possible on the command line (although there are similar metaphores available in modern X11 desktop environments like Gnome and KDE). This article tries to give you some background into, er, background processes.

To get a better feel for how to use job control, a brief look at how Unix handles processes can be helpful. As was mentioned in the opening section, Unix systems normally are running many processes at once. A process is defined as a program that is executing in memory, as opposed to an executable file (i.e., the program) that is sitting on the filesystem. When you log into a Unix system, you are running some shell program (e.g., tcsh or bash). When you ask the shell to run another program, such as vi, a new process starts and takes over the terminal from the shell. That new process is in the foreground by default. When you type commands, it is vi that responds, not the shell. When you exit vi, that process ends and parent process, the shell, returns. When you run vi, the shell itself goes into the background. You've been using background processes all along.

You may have noticed that I slipped in a new concept about processes in the last paragraph. Process are related to each other in hierarchical way by the kernel. When you execute a command from the shell, that new command is a child process of the shell. When a process terminates, the parent process is notified and is given an opportunity to take some action. What happens when you log out? All your shell's child processes are terminated along with the shell process itself, and your system's getty daemon waits for a new user to log in. What happens when getty dies? The ultimate ancestor for all system processes on a Unix system is

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

So far, you've seen how to get processes into and out of the background. That's a pretty good start, but what happens when you put more than one process in the background? How do you remember what's in the background at all? Fortunately the jobs command, built into Bourne and C shell derviates, lists all your current session's background jobs. Let's see this in action. In the example below, I started several web browsers:

[jjohn@marian jjohn]$ jobs

[1]   Running                 netscape &

[2]-  Stopped                 lynx

[3]+  Stopped                 lynx http://aliensaliensaliens.com

Every background process is assigned a job number by your shell. This number is unique only for your current session. It isn't globally unique like a process ID. In fact, one job number is assigned to processes that are pipelined together. For example, the following line gets only one job number.

$ uniq bigfile.dat | sort | wc -l &

In the jobs example above, the first process was started with an ampersand, so it was immediately backgrounded. Job 2 started as a typical interactive session, but I stopped it with CTRL-z. A stopped process is not the same as a terminated process — it simply doesn't receive any CPU time. It's like a caveman frozen in ice, waiting to be thawed out and come back to life. If you find that a job is becoming a resource hog, consider using CTRL-z to suspend the process until you figure out why it's being so gluttonous. The next job listed is a new instance of lynx, which is also put into the background so that the jobs command could be run for this listing. The plus sign next to the job number indicates that that job will be in the foreground when fg is typed. That job is known as the current job . The minus sign indicates the previous job , the job that used to be the current job.

Job numbers can be supplied to fg . In the given example, the first version of lynx can be revived using fg %2. You can also kill jobs with the job number. Why have two versions of

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're using Bourne-type shells, you have to watch out for putting a series of commands separated by semicolons (Section 28.16) into the background. These shells put only the last command on the line into the background, but wait for the first.
An easy way to test this is with the following command line, which waits for 15 seconds, then does an ls:
```
$ sleep 15; ls &

                  
```
In Bourne-like shells, you won't get your prompt back until the sleep (Section 25.9) command has finished.
With Bourne-type shells, the proper way to put a series of commands into the background is to group them with parentheses:
( ) Section 43.7
```
$ (sleep 15; ls)&

                  
```
This may strike you as a defect, but in fact, it's a sign of the greater precision of Bourne shell syntax, which makes it somewhat exasperating for interactive use but much better for programming.
It doesn't make any sense to run an interactive program such as an editor in the background. For example, if you type this from the C shell:
```
% vi &

[1] 3071
```
you'll get a message like the following:
```
[1]  + Stopped (tty output) vi
```
vi can be active only in the foreground. However, it does make sense to have vi stopped (Section 23.1) in the background.
If you are running vi or any other interactive program, you can quickly get back to the shell by typing CTRL-z to stop the program. The shell will take control of your terminal and print another shell prompt.
Stopping vi (Section 23.6) is more efficient than using its shell escape mechanism (Section 17.21), since it lets you go back to your original shell rather than starting a new one. Simply type fg to get back to where you were in editing.
We have had the misfortune to share a system with new users who were overenthusiastic in their use of background processes, rather like the man who loved loving so much he sought many lovers. Because each background process is competing for the same resources, running many of them can be a drain on the system, and everything takes longer for everyone. We knew people who thought that if they ran three

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

% is the current stopped or background job, but not always the last one. If you've stopped any jobs, the current job is the most recently stopped job. Otherwise, it's the most recent background job. For example, try stopping your editor (like vi), then putting another job in the background:

% vi afile

CTRL-z

Stopped

% sleep 1000 &

[2] 12345

% fg

and notice that the fg brings your editor to the foreground.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I see too many people using a series of commands like the ones that follow. Programmers do this when they write and compile programs. Writers use this when they're making a draft file and running it through the formatter. They're probably wasting a lot of time and effort:

% vi somefile

               ...Edit somefile, then quit vi...

% someprog somefile

               ...Process somefile...

% vi somefile

               ...Edit somefile again...

% someprog somefile

               ...Process somefile again...

Each time they restart vi, they have to reset options and move the cursor to the place they were working before. After they restart, vi has forgotten the previous search (the n command), the previous action (the . command), the previous regular expression, the named and numbered buffers...

In the same way, why quit any other program (that isn't an editor) if you aren't done with it? The programs lose their state. For instance, quitting a man (Section 2.1) or info (Section 2.9) command when you're in the middle of a document means that when you start it again, it'll be at the start. It will have forgotten the last term you searched for.

If your system has job control (Section 23.1), that solves all these problems. (If it doesn't, you can still use a shell escape (Section 17.21).) Instead of quitting vi, get into command mode and write your buffer with the :w command. Stop the editor with the CTRL-z command. Then process the file. When you're ready to do more editing, bring your vi job back into the foreground with fg. The editor will be just where it was.

Even better, you can set vi's autowrite option. If you've made any changes to the buffer before you press CTRL-z, vi will automatically write the buffer. You won't need to remember to type :w before you stop the editor. You can set autowrite at a colon (:) prompt, but I set it in my .exrc file (Section 17.5) instead.

You don't absolutely have to write your file before suspending vi. It's a good piece of advice, but not required by the job control mechanism. Typing CTRL-z will suspend the editor whether you've written out your files or not.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If your computer is barely crawling, you can kill ( Section 24.12) some processes, but you'll have to start them again later. On many Unix systems, you can renice ( Section 26.7) the processes, but you won't be able to raise the priority again later, after the system speeds up, unless you're the superuser (Section 1.18).

If you don't need your results right away (and you won't get them, anyway, if the system is crawling!), try stopping some jobs. The best candidates are "CPU-eaters" like formatters, compilers, and any job that runs up a lot of time quickly in the ps (Section 24.5) or time (Section 26.2) reports. Start them again later, and the jobs will take up where they left off.

If the job is in the foreground, just press CTRL-z (Section 23.3) to stop it. If the job is running in the background and you're running csh or tcsh, use the shell's stop command with a job identifier — for example, stop %3 or stop %cc.

On other shells — even shells without job control (!) — you can use kill (Section 24.12) with the -STOP signal and either the job number or process ID number. The csh and tcsh command stop does this for you. On other shells, if you'd like, you can add an alias named stop to the shell setup file (Section 3.3). Later, when the system speeds up, put the job back into the background with bg or into the foreground with fg. For example:

bash$ alias stop='kill -STOP'

bash$ jobs

[1]+  Running                 g++ hugeprog.cc &

bash$ stop %1

[1]+  Stopped (signal)        g++ hugeprog.cc

    ...later...

bash$ bg %1

[1]+ g++ hugeprog.cc &

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Normally, the shell tells you about changes to your background jobs whenever it prints its prompt. That is, when you do something that makes the shell give you a prompt, you'll get a message like:

[1]  + Stopped (tty input)   rm -r

%

This message tells you that the rm -r command, which you're running in the background, needs input; it has probably asked you whether or not to delete a read-only file, or something similar.

This default behavior is usually what you want. By waiting until it prints a prompt, the shell minimizes "damage" to your screen. If you want to be notified immediately when a job changes state, you should set the variable notify:

% set notify      

               ...csh, tcsh

$ set -o notify   

               ...bash, ksh

$ setopt notify   

               ...zsh

The drawback, of course, is that you may be analyzing a screenful of output that you've laboriously constructed, only to have that screen "destroyed" by a lot of messages from the shell. Therefore, most users prefer to leave notify off (unset). To stop all background output, use stty tostop (Section 23.9).

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you put a job in the background and don't redirect (Section 43.1) its output, text that the job writes to its standard output and standard error comes to your screen. Those messages can mess up the screen while you're using another program. You could lose the (maybe important) messages, too — they might scroll off your screen and be lost, or your foreground program may clear the screen and erase them.

Many Unix systems have the command stty tostop. Type that command at a prompt, or put it in your .login or .profile file. After that, your shell's background jobs that try to write to your terminal will be stopped. When you want to see the background job's output, bring it into the foreground (with fg).

How will you know that the background job has been stopped? The shell will print a message like this just before it prints a prompt:

[1] + Stopped (tty output)    somejob

%

The shell can also interrupt your foreground job with that message as soon as the background job is stopped. To make it do that, set notify (Section 23.8).

In C shell, you can turn off this feature and let background jobs write to your terminal any time with the command:

% stty -tostop

In bash , the command is similar:

$ stty tostop

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When Unix first started, even local terminals very often communicated with the system via short-haul modems. (After all, Unix was invented by the phone company.) When someone logged out, the modem hung up the phone — and conversely, if the modem hung up, a "hangup" signal was sent to the login shell, whereupon it terminated, bringing down all its child processes (Section 24.3) with it.

In the C shell, processes that you run in the background are immune to hangups, but in the Bourne shell, a process that you started in the background might be abruptly terminated.

Go to http://examples.oreilly.com/upt3 for more information on: nohup

The nohup command ("no hangup") allows you to circumvent this. (The GNU version is on the web site.) Simply type:

$ nohup command &

Any output from command that would normally go to the terminal (i.e., has not been redirected) goes to a file named nohup.out in the current directory.

Of course, if you want to run jobs at off hours, you might do even better using at, cron, or batch.

nohup is sometimes handy in shell scripts to make them ignore the HUP and TERM signals ( Section 24.10), though trap (Section 35.17) is more versatile. (In System V, nohup causes a command to ignore HUP and QUIT, but not TERM.)

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Job control isn't always a good thing. For instance, I might want to start a long equipment-monitoring job running when I go home for the night. But if I simply put the job in the background and try to log out, zsh says

zsh: you have

running

jobs. If I log out anyway, the shell sends my background job a HUP signal. I could use nohup (Section 23.10) to block the hangup signal, but there's a simpler way: tell the shell, "Don't use job control on this job." This is also true of jobs that I know are there — a clock running on my X Window System display, for instance — and that I'll never want to use job control on, so the jobs are just cluttering the jobs (Section 23.3) list.

To run a job without job control, the trick in most shells is to start the job in a subshell (Section 43.7), and put the job inside that subshell into the background. This is sometimes called "disowning" the job. Note that the ampersand (&) is inside the parentheses:

% (myprog -opts &)

The job won't appear in the jobs list, but ps ( Section 24.5) should show it running. (You might need to use a "show all jobs" option like ps -x or ps -e.) If you use ps -l for a "long" listing, you'll see that the process' PPID (the process ID number of the parent process (Section 24.3)) is 1; this means that the process is now "owned" by init (Section 24.2). On the other hand, if you'd started the job in the background normally (without the subshell trick), you'd see that its PPID was that of the shell you started it from.

The Z shell has a more direct way: its &! and &| background operators. Both of them do the same thing: if you use one of those operators instead of plain &, the job will be disowned immediately; it won't appear in the jobs list.

In most shells, once you start a job without the subshell trick, the shell that started the job will continue to be its parent. (Some shells, like the C shells, will give up ownership of a child process and let it keep running when you end the shell — that is, when you log out — and then

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Your Linux workstation display may look like just one terminal. It's actually seven terminals — or even more — in one. Linux has built-in virtual consoles, a series of ttys (Section 2.7) that you can log into separately: each one can have a login session, with its own shell, working at the same time as the others. You can see only one of these consoles at once; you bring a console into view by pressing a hot-key combination. For instance, I log into the first virtual console as root and the second as myself.

If your Linux system comes up after a reboot with a mostly blank screen something like this:

Red Hat Linux release 6.2 (Zoot)

Kernel 2.2.14-5.0 on an i686



penguin login:

you're seeing one of the virtual consoles — in this case, it's the first one you've seen since the reboot, so it has to be console number 1. On the other hand, if your system boots to an X Window display with a graphical xdm or gdm login box, you're using a different virtual console, probably console number 7. All of this is configurable. But by default, consoles 1 through 6 are ttys, with getty ( Section 24.2) processes running, ready to manage individual login sessions. Virtual console 7 is an X Window System display.

To switch between the consoles — to bring a differnt console "to the front" — use the hot-key combination CTRL-ALT-n, where n is the console number. (Actually, the only time you need the CTRL key is when the X Window console is in front. When you've got a nongraphical console in front, you can switch with just ALT-n. But if you find the difference hard to remember, there's no problem with always using the CTRL key.)

Here's one of the reasons I like to start my window system by typing a command (startx) at a shell prompt in a virtual console. The X server, and client applications running under X, will spit error messages onto the standard output (or standard error) at the console where I ran startx. So it's easy for me to jump back to the console — by pressing CTRL-ALT-2 — to see error messages. Then I can jump back to X with CTRL-ALT-7. (I actually changed this setup, later, to log X errors to a file that I watch from a window, but that's another story.)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Once you start telnet, rlogin, rsh , and ssh for an interactive login, they basically "take over" your shell. Your keystrokes are sent to the shell running on the remote system. So if you type CTRL-z, it won't stop the telnet (or whatever) job: it'll stop the job running on the remote system.

It can be very handy to suspend a connection to a remote system and resume it, sometime later, with fg. Most of the remote login utilities let you do that.

To stop a telnet session, start by pressing the escape character. By default, this is CTRL-] (Control-right bracket). You should get a telnet command prompt. Type z to suspend the job, Here's how that looks:

myhost$ telnet remhost

Trying 198.59.115.17...

Connected to remhost.

Escape character is '^]'.



SunOS 5.6



login: whoever

   ...

remhost% CTRL]

telnet> z



[1]+  Stopped                 telnet remhost



myhost$

You can use other commands at the telnet command prompt. For a list, see your manual page or type help at the prompt. If you get to that prompt accidentally and don't want to stop the telnet session, simply press ENTER once.

Other remote-login utilities don't have a command prompt. Their control commands start with ENTER-tilde (~) and one more control character. The command to stop the session is ENTER, then tilde, then CTRL-z. It won't appear on your screen as you type it (if it does appear, it didn't work . . . try again). For example:

myhost$ ssh remhost

Last login: Fri Dec 22 09:08:31 2000 from myhost

NetBSD 1.4.2A (GENERIC) #6: Wed May 31 06:12:46 EST 2000



remhost%

remhost% ~CTRL-z



[1]+  Stopped                 ssh remhost



myhost$

Notice the extra prompt: it shows me pressing ENTER first, before typing the tilde. That isn't necessary if you pressed ENTER to complete the previous command line — but I tend to do it all the time, "just in case" I didn't type that ENTER.

You can stop the remote session in the middle of an interactive job, like using a text editor. But I'd recommend getting to a shell prompt on the remote system first, if you can. (For example, stop the remote job with CTRL-z so you'll get a shell prompt on the remote system.) Otherwise, if you bring the remote session into the foreground while you're in the middle of a full-screen editing job there, for example, the remote system won't know that it's supposed to redraw the screen when you come back online. Worse, if you forget where you were on the remote system, you might type a key that could do something disastrous, like deleting lines of the file you're editing. Stopping and starting from a known point — a shell prompt — is the best way I've found.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We've already talked about so many of the topics in this chapter, here or there, that it may seem like a real hodgepodge. It's a grab-bag of important things to know about processes — which you can think of as programs that are actually running, rather than sitting on the disk somewhere.

The chapter starts out with a couple of conceptual articles. They define some important terms that you're likely to encounter in this chapter.

Then we talk about the ps command, which tells you what processes you have running and just what they are up to (Section 24.5, Section 24.6, Section 24.8).

The next few articles cover signals, which are one way processes communicate with one another. We cover topics like:

What are signals (Section 24.10)?
How to send signals from the keyboard (Section 24.11 and Section 24.12; also see Section 5.8).
How shell programs can "handle" signals (Section 24.13 and Section 35.17).

We go from there to a more general discussion of ways to kill processes:

How to kill all your processes (Section 24.14).
How to kill processes by name rather than by process ID (Section 24.16).
How to stop runaway jobs (Section 24.17).
Why some processes don't seem to go away when you kill them (Section 24.18, Section 24.19).
How to get rid of a frozen window (Section 24.22).
How to make sure processes don't die when you log out (Section 23.10).

— TOR

We discuss fork and exec in Section 27.2, but the concept comes up so often in this chapter that we thought we ought to have a closer cross reference.

Put simply, fork and exec are the Unix system calls (requests for operating system services) that Unix programs use to create new processes. When you start up a Unix system, it starts with only one process, a program called init.

How does init magically turn into the hundreds or perhaps even thousands of processes that make up a working Unix system? That's where fork and exec come in.

One process spawns another ("spawn" is another term you should get used to seeing) either by replacing itself when it's done — an

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We've already talked about so many of the topics in this chapter, here or there, that it may seem like a real hodgepodge. It's a grab-bag of important things to know about processes — which you can think of as programs that are actually running, rather than sitting on the disk somewhere.

The chapter starts out with a couple of conceptual articles. They define some important terms that you're likely to encounter in this chapter.

Then we talk about the ps command, which tells you what processes you have running and just what they are up to (Section 24.5, Section 24.6, Section 24.8).

The next few articles cover signals, which are one way processes communicate with one another. We cover topics like:

What are signals (Section 24.10)?
How to send signals from the keyboard (Section 24.11 and Section 24.12; also see Section 5.8).
How shell programs can "handle" signals (Section 24.13 and Section 35.17).

We go from there to a more general discussion of ways to kill processes:

How to kill all your processes (Section 24.14).
How to kill processes by name rather than by process ID (Section 24.16).
How to stop runaway jobs (Section 24.17).
Why some processes don't seem to go away when you kill them (Section 24.18, Section 24.19).
How to get rid of a frozen window (Section 24.22).
How to make sure processes don't die when you log out (Section 23.10).

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We discuss fork and exec in Section 27.2, but the concept comes up so often in this chapter that we thought we ought to have a closer cross reference.

Put simply, fork and exec are the Unix system calls (requests for operating system services) that Unix programs use to create new processes. When you start up a Unix system, it starts with only one process, a program called init.

How does init magically turn into the hundreds or perhaps even thousands of processes that make up a working Unix system? That's where fork and exec come in.

One process spawns another ("spawn" is another term you should get used to seeing) either by replacing itself when it's done — an exec — or, if it needs to stay around, by making a copy of itself — a fork. In the latter case, the forked copy commits polite suicide by execing the desired second program.

A good example of this whole sequence can be seen in the way a Unix system's login procedure for terminals (non-network (Section 1.21) logins) works. The init process spawns a series of getty processes, each of which monitors a serial port (a tty), looking for activity. It's the getty program that actually puts up the first login: prompt.

Once someone actually types a login name, getty's job is done; it execs the login command. login prompts for a password (if the account has one) and, if the password is okay, execs the login shell. Whenever you start another program, the shell forks itself, and the copy execs whatever program you asked to run.

That's why some commands are built into the shell (Section 1.9). There's overhead involved in starting a new process. What's more, because a child process can't affect its parent's environment (Section 24.3), some commands don't make sense as separate processes. For example, cd must be built in, or it couldn't change the working directory for the current shell.

There's an exec command that you can type at a shell prompt; see Section 36.5. Watch out, though: it will replace your shell with whatever command you

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As you know, when you log into your Unix account and start typing, you're talking to the shell (Section 27.1). The shell you use may be a variant of the Bourne shell (such as a standard sh, ksh, or the GNU shell bash), or perhaps it is a variant of the C shell, csh (such as, perhaps, the tcsh shell that includes line- and history-editing features). Alternatively, you may be using a somewhat less common shell such as rc.

Your shell is a process, one of many individual programs running at the same time on the machine. Every process has certain pieces of information associated with it, including the following:

The process ID (PID) is a number assigned to the process when it is started up. Process IDs are unique (that is, they cycle and are eventually reused, but no two processes have the same process ID at the same time).
The user ID (UID) tells who the process belongs to. This determines what files and directories the process is allowed to read from or write to (Section 50.1), as well as who is allowed to kill the process (Section 24.12) (tell it to stop running).
The group ID (GID) is similar to the user ID but tells which group the process belongs to. On some systems, this controls the group assigned to files created by the process. See Section 50.2.
The environment contains a list of variable names and associated values. For example, when you type echo $HOME at the shell and it prints out the name of your home directory (Section 1.15), it has told you the contents of the environment variable (Section 35.3) called HOME.
The current working directory (Section 31.3) is the directory that is currently the default. When you specify a filename to a program but do not say explicitly where to look for it with a

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In Unix, when a program starts another program (more exactly, when a process starts another process), the new process runs as a subprocess (Section 24.3) or child process. When a shell starts another shell, the new shell is called a subshell.

So what? There are some important things to know about it: the child process gets a copy of its parent's environment, and any changes in the environment of the child process aren't passed to its parent. "Still," I hear you say, "so what??"

Shell scripts are run in a subshell (unless you use the source or . commands (Section 35.29) to start the script). If the script makes changes to the environment of its (sub)shell, the parent shell won't see those changes. If the script uses cd, it doesn't change the current directory in the parent shell. If the script changes the value of the TZ (or any) environment variable, that won't change TZ in the parent shell. The script can set a different umask than the parent shell — no problem.
There are times you might want to start a subshell from your current shell. Maybe you have a special project where you need to work in a different current directory, reset environment variables, set a new home directory, reset some aliases, use a different PATH (Section 35.6), whatever. When you end the subshell, the parent shell's environment will be the way it was.
If your parent shell has job control (Section 23.3), you can stop the subshell and pop back to your parent shell without losing the changes in the subshell. If the child shell has job control, too, the command suspend (or kill -STOP $$ (Section 27.17)) will stop it. Otherwise, just type CTRL-z at the subshell's prompt. For example:
prompt Section 4.1
```
myprompt% csh

myprompt% set prompt="project% "

project% cd project-directory

                     

project% setenv PRINTER plotter

project% set path=($path some-new-directories)

project% setenv EXINIT "se ts=4 sw=4 aw wm=0"
```

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The ps command varies from system to system. (The ps on one Red Hat Linux system reads a PS_PERSONALITY environment variable with 21 possible settings!) This article describes several different versions. Yours is probably different in some ways, so check your ps manual page for details.

The ps command produces a report summarizing execution statistics for current processes. The bare ps command lists the process ID, the terminal from which the command was started, how much CPU time it has used, and the command itself. The output looks something like this (it differs by system):

  PID TT STAT  TIME COMMAND

 1803 p5 IW    0:00 -csh (csh)

 1883 p5 IW    0:04 vi outline

 1811 p6 IW    0:01 -csh (csh)

 5353 p6 TW    0:01 vi 4890

By default, ps lists only your own processes. There are many times, though, when it's desirable to have a more complete listing with a lot of data about all of the processes currently running on the system. The options required to do this differ between BSD Unix and System V. Under BSD Unix, the command is ps -aux, which produces a table of all processes, arranged in order of decreasing CPU usage at the moment when the ps command was executed. [The -a option gives processes belonging to all users, -u gives a more detailed listing, and -x includes processes that no longer have a controlling terminal (Section 24.6). — TOR] It is often useful to pipe this output to head (Section 12.12), which will display the most active processes:

% ps -aux | head -5

USER       PID %CPU %MEM   SZ  RSS TTY STAT  TIME COMMAND

martin   12923 74.2 22.5  223  376 p5  R     2:12 f77 -o foo foo.F

chavez   16725 10.9 50.8 1146 1826 p6  R N  56:04 g94 HgO.dat

ng       17026  3.5  1.2  354  240 co  I     0:19 vi benzene.txt

gull      7997  0.2  0.3  142   46 p3  S     0:04 csh

The meanings of the fields in this output (as well as others displayed by the -l option to ps) are given in Table 24-1.

The first line of this output shows that user martin is running a FORTRAN compilation (f77). This process has PID (Section 24.3) 12923 and is currently either running or runnable. User

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In Section 24.5, we pointed out that the ps command needs special options (-x for BSD-derived versions and -e for System V-type) to list processes without a controlling terminal.

But just what is a controlling terminal? Just what it sounds like: the terminal from which the process was started. In the ps listing, this is usually given as a tty, or terminal ID. That ps entry usually corresponds to a serial port, or a pty. A pty or "pseudo-terminal" is a construct that makes a window or network login (Section 1.21) look to the operating system just like a terminal.

In the ps listing, a tty might appear as t1 for /dev/tty1, p3 for /dev/ttyp3, or as some other designation, such as co for /dev/console, the full-screen display of a workstation before any window system is started. Processes without a controlling terminal show a question mark (?).

How does a process "lose" its controlling terminal? Easy. Some processes, such as system "daemons" (Section 1.10) never had one — they were started by system scripts that weren't started from any terminal, or they disconnected themselves from their controlling terminals. But it's also possible that you started a process running in the background, logged out, and logged back on later or on another terminal to find it still running without a controlling terminal. Disowned processes (Section 23.11) fit this category too.

The tty command can be used to report which "terminal" you're currently connected to. For example:

% tty

/dev/ttyp2

Running tty without a controlling terminal gives the message not a tty.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

ps without arguments lists all processes started from the current terminal or pseudo-terminal. But since ps is not a shell command, it doesn't correlate process IDs with the shell's job numbers. It also doesn't help you find the ID of the runaway process in another shell window.

To get this information, use ps -a (for "all"); this lists information on a different set of processes, depending on your Unix version.

Instead of listing all that were started under a specific terminal, ps -a on System V-derived systems lists all processes associated with any terminal that aren't group leaders. For our purposes, a "group leader" is the parent shell of a terminal or window. Therefore, if you are using a windowing system, ps -a lists all jobs started in all windows (by all users), but not their parent shells.

Assume that, in the previous example, you have only one terminal or window. Then ps -a will print the same output as plain ps except for the first line, since that's the parent shell. This doesn't seem to be very useful.

But consider what happens when you have multiple windows open. Let's say you have three windows, all running terminal emulators such as xterm for the X Window System. You start background jobs alice, duchess, and hatter in windows with pseudo-terminal numbers 1, 2, and 3, respectively. This situation is shown in Figure 24-1.

Figure 24-1: Background jobs in multiple windows

Assume you are in the uppermost window. If you type ps, you will see something like this:

   PID TTY      TIME COMD

   146 pts/1    0:03 bash

  2349 pts/1    0:03 alice

  2390 pts/1    0:00 ps

But if you type ps -a, you will see this:

   PID TTY      TIME COMD

   146 pts/1    0:03 bash

  2349 pts/1    0:03 alice

  2367 pts/2    0:17 duchess

  2389 pts/3    0:09 hatter

  2390 pts/1    0:00 ps

Now you should see how ps -a can help you track down and kill (Section 24.12) a runaway process. If it's hatter, you can type kill 2389. If that doesn't work, try kill

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There is a reason that some versions of ps, and thus derivatives such as w, sometimes print commands in parentheses:

% ps -f -u jerry

     UID   PID  PPID  C    STIME TTY      TIME COMMAND

   jerry 29240 29235  0 07:56:19 ttyp1    0:01 sh find_mh_dupes

   jerry 29259 29240 23 07:57:52 ttyp1    0:07 (egrep)

The reason is that whoever wrote ps liked it that way. The parentheses indicate that the command overwrote its name, or ps could not find the name, and that ps is printing instead the "accounting name." (The accounting name is the last component of the name given to the exec (Section 24.2) system call, and is the name used in the system resource usage accounting file.) Basically, ps does this in the C language:

if (proc->argv == NULL || strcmp(proc->acct_name, proc->argv[0]) != 0)

   printf("(%s)", proc->acct_name);

In the case of a large environment, ps is unable to find the argument vector. This is because it reads only the last few stack pages of each process.

Other versions of ps use completely different mechanisms for locating the command arguments and may never print parentheses.

— CT, in net.unix-wizards on Usenet, 13 November 1983

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In Unix, it seems almost everything can be treated like a file (Section 1.19). On many modern Unix systems, even processes are files — well, sort of. A special filesystem named /proc doesn't actually "contain" processes, but it lets you interact with them. Almost all of the "files" in /proc are plain text, so you can access them from scripts and programs, as well as from the command line. Of the systems I've checked, my Red Hat Linux 6.2 box (kernel version 2.2) seems to have the most in /proc, so I'll cover it. Please check your documentation — a proc(5) manual page, for instance — for the story on your system.

All /proc filesystems have one subdirectory for each process currently running on the system. Each of those process subdirectories is named for its PID (Section 24.3). Some versions of /proc also have other named files and subdirectories — and my system has a lot of them. Here's a partial listing of my /proc filesystem at the moment; I've left out a lot of the numbered subdirectories:

-F Section 8.10

$ ls -F /proc

1/        17415/    467/      cmdline      ksyms        pci

1047/     2/        482/      cpuinfo      loadavg      rtc

1052/     3/        5/        devices      locks        scsi/

1057/     345/      553/      dma          mdstat       self@

1287/     370/      593/      fb           meminfo      slabinfo

1289/     379/      594/      filesystems  misc         stat

14288/    393/      595/      fs/          modules      swaps

14289/    4/        596/      ide/         mounts       sys/

17409/    4017/     597/      interrupts   mtrr         tty/

17412/    407/      6/        ioports      net/         uptime

17413/    425/      apm       kcore        partitions   version

17414/    439/      bus/      kmsg

Linux system utilities like ps and pidof use information from /proc. Your programs can use it, too; there are some examples below. But it's also useful when you want to know something about your system. The "files" in /proc are most useful there. Let's look at a series of examples. We'll end with the numbered per-process "directories."

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Signals are a simple but important means of interprocess communication. Interprocess communication sounds fancy, but it's really a simple concept: it's the means by which one program sends a message to another program. It's common to think of signals as special messages sent by the Unix kernel (Section 1.10) but, in fact, any program can signal any other program.

What kinds of messages can you send with a signal? Relatively few, in reality. Signals aren't "arbitrary" messages, like letters; they are a small group of pre-defined messages, each with its own special meaning. System V Unix supports 16 signals, each of which is assigned a number; BSD-derived Unix implementations and SVR4 have 32 signals. Table 24-2 lists some of the more commonly used signals. It also lists keyboard characters that send common signals on BSD systems (these can be changed; see Section 5.8).

Table 24-2: Common signals
Signal name	Number	Meaning and typical use
HUP	1	Hangup — stop running. Sent when you log out or disconnect a modem.
INT	2	Interrupt — stop running. Sent when you type CTRL-c.
QUIT	3	Quit — stop running (and dump core). Sent when you type CTRL-\.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You probably know that typing CTRL-c ( Section 24.10) will terminate your foreground job. But what actually happens when you type CTRL-c?

When you type CTRL-c, you're sending the INT (interrupt) signal (Section 24.10) to the foreground process. Most well-designed programs "catch" the interrupt signal, which means that the program installs some special function (a "signal handler") that is called whenever a signal arrives. The signal handler normally closes all open files, resets your terminal properly (if needed), and does anything else necessary so that the program can depart from this world in peace. Then the program terminates. The QUIT signal, sent by CTRL-\, works similarly but also makes a core file for debugging.

Of course, it's possible for the signal handler to do something else entirely: the program can decide not to quit, or it can implement some truly bizarre feature. In fact, editors such as vi or Emacs almost always ignore most signals. The trap ( Section 35.17) command handles signals in the Bourne shell.

Whenever you send a signal from the keyboard, it's sent to all processes in the same process group ( Section 24.3). This may include the program's child processes, but it may not. And, of course, child processes can choose to ignore signals on their own. But more often than not, killing the parent process kills its children.

Section 5.8 explains how to set the key that sends these and other signals. The kill (Section 24.12) command also sends signals.

—ML and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sometimes it's necessary to eliminate a process entirely or to signal a process (Section 24.13); this is the purpose of the kill command. You can use the kill command with or without a signal id:

% kill 

               pid

% kill 

               -signal pid

where pid is the process' identification number, and signal (which is optional) is the signal to send to the process. The default signal is number 15, the TERM signal, which tells the process to terminate. On some systems, the signal must be specified numerically; others allow you to use either the signal number or its symbolic name. [Use kill -l for a list of signal names; unfortunately, the listing doesn't show the correspondence of names and numbers. However, they are in order, so if you can count, you can figure it out. — TOR]

Sometimes, a process may still exist after a kill command. If this happens, execute the kill command with the -KILL or -9 option. This almost always guarantees that the process will be destroyed. However, it does not allow the dying process to clean up, and therefore may leave the process' files in an inconsistent state.

Occasionally, processes will not die even after being sent the KILL signal. The vast majority of such processes fall into one of three categories:

Zombies. A process in the zombie state (Section 24.19) is displayed as Z status in BSD ps ( Section 24.5) displays and as <defunct> under System V. When a process is exiting, it informs its parent of its imminent death; when it receives an acknowledgment, its PID is removed from the process table. A zombie process is one whose total resources have been freed, but whose parent process' acknowledgment has not occurred. Usually, init will step in when the parent is gone, but very occasionally this fails to happen. Zombies are always cleared the next time the system is booted and do not adversely affect system performance.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[This article may not appear to have a lot to do with the subject of this chapter, but it illustrates the other side of signal handling — what a program or shell script can do when it receives a signal. Jerry's script uses the trap ( Section 35.17) command to catch several different signals and act differently depending on whether the signal is a "hangup" (HUP , or signal 1) or a TERM (signal 15). — TOR]

Unix systems run "daemon" programs such as cron(8) and syslogd(8) that wait in the background, looking for work to do. Many daemons read configuration files when they start up. System administrators sometimes change the configuration files and want the daemon to reread the file. One way to do that is by terminating and restarting the program — but that's ugly and also means the daemon won't be running for a few seconds until it's restarted. So many daemons are designed to reread their configuration files and/or restart themselves when they get a signal (usually the HUP signal, signal 1). System administrators do this by getting the daemon's process ID number and sending the signal with the kill command. Because the daemon "catches" the signal, the daemon isn't actually killed.

You can run a shell script as a daemon by putting it in the background. Here's a simple example, a shell script named watchq. It reads a file full of printer queue names and stores it in a shell variable. Every 30 seconds, it runs lpq ( Section 45.2) on all printer queues listed. If any queues have an error, the script echoes a message and the output of lpq to a particular user with the write (Section 1.21) command. (You could change it to write to the system's syslog by calling logger(1) instead of write. Or use xmessage (Section 36.26) to pop a notice window onto someone's X Window System console. Etc., etc.)

The script uses numbers (0, 1, 15) instead of signal names (EXIT, HUP, TERM). This is for portability to older Unix shells that don't understand names in

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

On many Unix systems, kill (Section 24.12) interprets the special "process ID" -1 as a command to signal all your processes (all processes with your user ID), except for the process sending the signal. For example, the following command will terminate all your processes:

% kill -TERM -1

To see if your system supports this feature, type man 2 kill (Section 2.1) to read the kill(2) manual page.

You can use this to prevent background jobs from continuing after you logout; just stick

kill -TERM

-1

into your .logout file. There are some good reasons not to do this though: if you use several terminals, this will kill all your processes when you log out from any terminal.

This command is also useful in desperate situations. If processes are spawning out of control, or if your terminal is locked, you can log in from another terminal and kill everything, without having to dig through ps (Section 24.5) to find the right process. The zap (Section 24.16) script searches process lists and kills processes automatically.

The special -1 process ID is defined differently for the superuser; if you're root, it means "all processes except system processes."

If you can't use the -1 process ID, and you use the Bourne shell or another shell without job control, you can use a 0 (zero) process ID. That sends the signal to all members of the process group (that is, processes resulting from the current login). A 0 doesn't work on shells, such as the C shell, that have job control (Section 23.3).

—ML, JP, and JIK

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This article discusses a particular version of kill that has some problems. Your system may have a different kill and, possibly, a version of killall (Section 24.16) that doesn't seem to have as many problems. But this article is worth reading anyway for what it shows you about process names and the ps command. It's good info to keep in mind when you're trying to kill processes in a hurry.

On my Linux system, the kill(1) manual page says I can send signals to processes either by PID numbers (as we showed in Section 24.12) or by process names. To an old stick-in-the-mud Unix user like me, who's been killing processes by their PIDs for 20 years, this doesn't seem very appealing. But hey, even I appreciate some of the new things Unix and Linux can do! ;-) So we're saying that, if my system is slow and I want to temporarily stop the two gcc compiles I'm running in the background, I can just type:

$ kill -STOP gcc

[2]   Stopped            gcc -c bigprog.c sub1.c sub2.c ...

[4]-  Stopped            gcc -o something something.c

Not necessarily. This is not always as simple as it seems. For one, before you kill a process by name, you'd better be sure that there are no other processes by that name, owned by you, running at the same time — unless you want to kill them too. That includes processes on other windows and ttys you're logged onto at the time; it also includes at, cron, or batch jobs that are running somewhere else on the system. Second, the process name may not be what you think it is. Third, even if your kill(1) manpage says that kill can do this, your shell may have a built-in kill that doesn't understand how to kill processes by name.

For example, let's say I have a runaway shell script named cruncher. I'm running it twice, and I want to kill both instances. Watch:

& Section 23.2

1$ cruncher & cruncher &

[1] 21451

[2] 21456

2$ kill cruncher

bash2: kill: cruncher: no such pid

3$ type -all kill

kill is a shell builtin

kill is /bin/kill

4$

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you want to kill processes, it's a pain in the neck to run ps (Section 24.5), figure out the process ID, and then kill the process — although sometimes you have to do it that way (Section 24.15). We'll look at two easier ways.

Many systems have a command named killall with a -i ("interactive") option. Be careful, though, because there are several versions, and the most basic does just what it says: kills all processes on the system (when run as the superuser (Section 1.18)). Check killall's manual page on your system.

The version of killall we're talking about here accepts multiple process-name arguments on its command line. Without its -i option, the command sends a signal (by default, TERM ) to any process name that matches. The process name you give has to match completely. Unfortunately, killall sends a signal to any process with that name — even processes owned by other users, which you can't kill (unless you're the superuser); you'll get the error

Operation

not permitted

. For example:

& Section 23.2, [5] Section 23.3

1$ cruncher & sleep 60 &

[5] 2714

[6] 2715

$ killall crunch eep

crunch: no process killed

eep: no process killed

$ killall cruncher sleep

sleep(2708): Operation not permitted

sleep(2710): Operation not permitted

sleep(2712): Operation not permitted

[5]   Terminated              cruncher

[6]   Terminated              sleep 60

With -i, cruncher lists the PID number and gives you a choice of typing y to kill a process or n to leave it alone:

$ cruncher & sleep 60 &

[5] 2732

[6] 2733

$ killall -i cruncher sleep

Kill sleep(2727) ? (y/n) y

sleep(2727): Operation not permitted

Kill cruncher(2732) ? (y/n) y

Kill sleep(2733) ? (y/n) y

Kill sleep(2734) ? (y/n) n

[5]   Terminated              cruncher

[6]   Terminated              sleep 60

A more flexible way to kill processes interactively is the zap shell script, presented by Brian Kernighan and Rob Pike in their classic book

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Especially if you're a programmer, you can run into a situation where you have processes forking (Section 24.2) out of control — more and more of them. By the time you kill one, fifty more fork.

On systems with job control ( Section 23.3), there's a good answer: use the STOP signal to stop the processes:
kill Section 24.12
```
kill -STOP ...
```
Stop any process you can so that it can't fork more processes. Stop them all. Then start cleaning up with kill -9.
If your system manager has set a per-user process limit on your computer, the good news is that your processes won't eventually crash the system. But the bad news is, when you try to run any command that isn't built into the shell (Section 1.9) (like killall (Section 24.16), which would be nice to use in this situation, if you have it):
```
% killall -STOP myprog

No more processes.
```
you can't because you're already at your limit.
If that happens, log on to another account or ask someone to run a command that will give a list of your processes. Depending on your system, the command is probably like one of these two:
```
% ps -u yourname 

                     

                     System V

% ps aux | grep yourname 

                     

                     BSD

                  
```
Then go back to your terminal and start stopping :-). If you get the No more processes error, your shell must not have a built-in kill command. Many shells do — including bash and csh. Carefully type the next commands to be sure that /bin/bash exists (assuming your shell has a built-in echo, this trick bypasses the external ls command); then, if the shell is there, replace your shell with bash. Don't make a mistake (if you do, you may not be able to log in again):
exec Section 36.5
```
$ echo /bin/bas?

/bin/bash

$ exec /bin/bash

bash$ kill ...
```

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You or another user might have a process that (according to ps (Section 24.5)) has been sleeping for several days, waiting for input. If you can't kill (Section 23.12) the process, even with kill -9, there may be a bug or some other problem.

These processes can be unkillable because they've made a request for a hardware device or network resource. Unix has put them to sleep at a very high priority and the event that they are waiting on hasn't happened (because of a network problem, for example). This causes all other signals to be held until the hardware event occurs. The signal sent by kill doesn't do any good.
If the problem is with a terminal and you can get to the back of the terminal or the back of the computer, try unplugging the line from the port. Also, try typing CTRL-q on the keyboard — if the user typed CTRL-s while getting a lot of output, this may free the process.
Ask your vendor if there's a special command to reset the device driver. If there isn't, you may have to reboot the computer.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Processes in your ps output that are in the <exiting> or Z status are called zombies.

You cannot kill zombies; they are already dead.

"What is a zombie?" I hear you ask. "Why should a dead process stay around?"

Dead processes stick around for two principal reasons. The lesser of these is that they provide a sort of "context" for closing open file descriptors (Section 24.3) and shutting down other resources (memory, swap space, and so forth). This generally happens immediately, and the process remains only for its major purpose: to hold on to its name and exit status (Section 35.12).

A process is named by its process ID or PID. Each process also has associated with it a parent process ID. The parent PID, or PPID, is the PID of the process that created it via fork (Section 24.2); if that particular process has since vanished, the parent PID is 1 (the PID of init (Section 24.2)). While the original parent is around, it can remember the PIDs of its children. These PIDs cannot be reused until the parent knows the children are done. The parent can also get a single byte of status (Section 35.12) from each child. The wait system call looks for a zombie child, then "collects" it, making its PID available and returning that status. The init(8) program is always waiting, so that once a parent exits, init will collect all its children as they exit and promptly ignore each status.

So, to get rid of a zombie, you must wait for it. If you have already done so or if the process' PPID is 1, the process is almost certainly stuck in a device driver close routine, and if it remains that way forever, the driver has a bug.

— CT

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Almost everything we cover in this book works as well from an old-style, full-screen terminal as it does from an terminal window (like xterm) under the X Window System (Section 1.22). Actually, a lot of it works on an old printing teletype, too! In all of those cases, you're interacting with a Unix shell. This article covers things you should know about using a shell from an X window. We'll talk specifically about the X11R6 xterm client, but this generally applies to any window with a shell inside of it — like GNOME terminal. This is a guided tour, so it helps to be at a workstation or other X display. If you can't take the tour, please scan through and look for the points I make along the way.

If you don't have an xterm window open, open one (by clicking on an icon, choosing a menu entry, or however you usually do it). We'll call this the "first window." Find its tty (Section 2.7). Next, in this first window, set an environment variable (Section 35.3) with a unique name and any value you want. You might call it FAVCOLOR and set the value to purple. Then, in that same window, type cd /tmp to change your current directory to /tmp. Finally, type

xterm -rv

-sb

(with no & after its name); this should open a second xterm window. Here's what that first xterm should look like (we'll show C shell syntax here):

% tty

/dev/pts/1

% setenv FAVCOLOR purple

% cd /tmp

% xterm -rv -sb

                  (cursor sits here; there's no shell prompt)

When your new second xterm pops open, it should be in reverse video (swapped foreground/background colors, the -rv option) to make it easy to identify, with a scrollbar too. In it, type tty to get its tty number, which will be different from the previous xterm's. Run env or printenv (Section 35.3), and you should see the special environment variable (like FAVCOLOR) that you set. Type pwd; the current directory should be /tmp.

If you've managed to follow this twisty series of steps, you've started a chain of processes

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

xterm is an X client that runs a Unix process on a pty "inside" a window. By default, this process is a shell: an instance of the same shell you log into the system with. But it can be basically any Unix process. As you saw in Section 24.20, when the process exits, the xterm window closes because its child process has gone.

To override the default shell process in an xterm window, use the -e option (Section 5.22), followed by the command line to run the process. This must be the last thing on the xterm command line. If you want to open an xterm window with no scrollbar (the +sb option) and with the vi editor in it, to edit the log file named logfile, run the command below:

% xterm +sb -e vi logfile

%

An xterm window should open with vi running inside it. If you don't know how to use vi, the best thing to do is to leave it alone until you've finished this example — then press the ESC key, type :q, and press ENTER to exit vi. When vi exits, its window should close too, and you'll get another shell prompt.

I chose to have you run vi in a window because the vi process keeps running until you tell it to quit, and then the window closes. Other Unix processes that don't wait for a "quit" command will terminate as soon as they're done, and the window closes before you can see the process output. For example, let's say you want to display a file in an xterm window with a scrollbar. Start by choosing a file and using wc -l (Section 16.6) to count the number of lines. Then open an xterm and a scrollbar, with the scrolling buffer length set to just the right number of lines:

cat Section 12.2

% wc -l somefile

     74 somefile

% xterm -sl 74 -sb -e cat somefile

%

What happened? Unless your window manager holds it there, the xterm window closes just after it opens. Why? Its child cat process exited, so the parent xterm did too. One easy answer is to use a shell that runs three commands. First is the command you want to run (here, cat). Next, echo a prompt. Finally, run the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In the X Window System, there's a process controlling every window. If the window (or its process) is frozen and you can't get rid of it, the easier way is usually to kill (Section 24.12) the process. As Section 24.20 explains, there may be a chain of processes running; the window could come from the parent process (as in the case of an xterm with a shell running inside of it) or it could be the child (such as when a shell script runs an X client like xmessage — as in the nup script below). Your job is to use ps (Section 24.5) to track down the process(es) behind the window and kill the right one(s). We'll look at two different examples, then look at a shell script that opens a window and, later, closes the window by killing its process.

Let's say you're running vi in an xterm window, and the window seems to be frozen. Start with some detective work: open up another xterm window and run ps alwx or

ps

-ef

. (If you're sure that all the processes in the window are owned by you — and none were set user ID (Section 1.17) — you can use run

ps

lwx

, for example.) You want a listing that shows the chain of process IDs, parent-to-child, in that window. The tty (Section 2.7) of the shell inside the xterm will help you find the right one, if you know it. For example, I found

vi

0568.sgm

running on the tty pts/5, so the shell I want (the parent of vi) must also be on pts/5. From the shell's parent ID, I can find the PID of the xterm that started the shell. (I'll cut some of the columns in this listing to make it easier to read.)

% ps alwx

 UID   PID  PPID  STAT TTY    TIME  COMMAND

1000 11287     1  S    tty2   0:44  xterm -sb -sl 2000

   ...

1000 11289 11287  S    pts/5  0:04  bash2

   ...

1000  2621 11289  S    pts/5  0:00  vi 0568.sgm

A Unix system cycles its PIDs. A child process may have a lower PID than its parent! (Here, vi's PID is 2621, but its parent's PID is 11289.)

Now you need to decide what process to kill. You could simply kill them all, assuming you own them (on some systems, the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you are more familiar with desktop systems than Unix, the concept of delayed execution may be new to you. After all, the prime mover of all activity in the desktop metaphor is the user. In Unix, all kinds of processes start, execute, and report without any users on the system.

There are a few good reasons why you need to know about delayed execution. The first is that long, noninteractive jobs are best run when the fewest users are likely to be on the system. Humans find responsive systems desirable; processes aren't as likely to complain about getting sporadic CPU time. The second situation in which delayed execution is desirable is when a resource you need is only available at certain times. For instance, your group of local workstations create tar archives for the day's work, and you need to grab those files and copy them to tape. The third reason for delayed execution is when you need to push or pull information on a regular basis. This is the case with web masters who need to push their updated content to their production environment from their editing machine. The reverse may also hold true: you may need to collect Rich Site Summary files from a variety of web sites for a local cache. In all these cases, you need processes to start without you, like a band of relentless software robots.

This chapter covers the following techniques of delayed execution:

The venerable cron ( Section 25.2) system schedules process for regular, periodic execution. It is the most frequently used utility for running programs after hours.
For processes that only need to run once at some future date, the at ( Section 25.5) command is ideally suited.
For simple scripts that need to pause before continuing on, the sleep (Section 25.9) command is available.

— JJ

This article covers two different versions of

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you are more familiar with desktop systems than Unix, the concept of delayed execution may be new to you. After all, the prime mover of all activity in the desktop metaphor is the user. In Unix, all kinds of processes start, execute, and report without any users on the system.

There are a few good reasons why you need to know about delayed execution. The first is that long, noninteractive jobs are best run when the fewest users are likely to be on the system. Humans find responsive systems desirable; processes aren't as likely to complain about getting sporadic CPU time. The second situation in which delayed execution is desirable is when a resource you need is only available at certain times. For instance, your group of local workstations create tar archives for the day's work, and you need to grab those files and copy them to tape. The third reason for delayed execution is when you need to push or pull information on a regular basis. This is the case with web masters who need to push their updated content to their production environment from their editing machine. The reverse may also hold true: you may need to collect Rich Site Summary files from a variety of web sites for a local cache. In all these cases, you need processes to start without you, like a band of relentless software robots.

This chapter covers the following techniques of delayed execution:

The venerable cron ( Section 25.2) system schedules process for regular, periodic execution. It is the most frequently used utility for running programs after hours.
For processes that only need to run once at some future date, the at ( Section 25.5) command is ideally suited.
For simple scripts that need to pause before continuing on, the sleep (Section 25.9) command is available.

— JJ

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This article covers two different versions of cron. There are other versions around: Vixie cron, for instance, has some different features and is common in Linux distributions. A variation called anacron doesn't assume (as cron does) that the system is running 24 hours a day; it's especially nice on portable computers. Rather than trying to cover every flavor, this article has information on older, basic crons that should show you some of what to expect in whatever version you have.

cron allows you to schedule programs for periodic execution. For example, you can use cron to call rsync every hour to update your production web site with new articles or to perform any number of other tasks.

With redirection ( Section 43.1), cron can send program output to a log file or to any username via email.

cron jobs are run by a system program in an environment that's much different from your normal login sessions. The search path (Section 27.6) is usually shorter; you may need to use absolute pathnames for programs that aren't in standard system directories. Be careful about using command aliases, shell functions and variables, and other things that may not be set for you by the system.

The cron system is serviced by the cron daemon ( Section 1.10). What to run and when to run it are specified to cron by crontab entries, which are stored in the system's cron schedule. On older BSD systems, this consists of the files /usr/lib/crontab and /usr/lib/crontab.local; either file may be used to store crontab entries. Both are ASCII files and may be modified with any text editor. Since usually only root has access to these files, all cron scheduling must go through the system administrator. This can be either an advantage or a disadvantage, depending on the needs and personality of your site.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

For a good tip on silencing cron job mailings, see Section 25.6.

Most recent versions of Unix have a special command for maintaining the crontab file. To create a new crontab file, create a file containing the desired crontab entries. Then run the crontab command to install the file in the cron spool area. For example, if user chavez executes the command below, the file mycron will be installed as /usr/spool/cron/crontabs/chavez:

$ crontab mycron

If chavez had previously installed crontab entries, they will be replaced by those in mycron; thus, any current entries that chavez wishes to keep must also be present in mycron.

The -l option to crontab lists the current crontab entries, and redirecting its output to a file will allow them to be captured and edited:

$ crontab -l >mycron

$ vi mycron

$ crontab mycron

The -r option will remove all current crontab entries. Many versions of the crontab have an additional -e option that lets you directly edit your current crontab entries in a single step.

On original BSD-based Unix implementations, there is no separate crontab command, nor does each user get a personal crontab file. It does distinguish between " global" crontab entries (in /usr/lib/crontab) and "local" entries (in /usr/lib/crontab.local) — however, you have to edit these files directly, which will probably require you to become superuser. It's a good idea to collect personal and site-specific crontab entries in the crontab.local file.

— AF, from Essential System Administration (O'Reilly, 2002)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Since crontab entries must be a single line long, it's hard to include any standard input with them. Sure, you can use commands like:

0 22 * * * echo "It's 10PM; do you know where your children are?" | wall

but you can't use "here documents" and other methods of generating multiline input; they intrinsically take several lines.

To solve this problem, cron allows you to include standard input directly on the command line. If the command contains a percent sign (%), cron uses any text following the sign as standard input for cmd. Additional percent signs can be used to subdivide this text into lines. For example, the following crontab entry:

30 11 31 12 * /etc/wall%Happy New Year!%Let's make next year great!

runs the wall command at 11:30 a.m. on December 31, using the text:

Happy New Year!

Let's make next year great!

as standard input. [If you need a literal percent sign in your entry, for a command like date +%a, escape the percent sign with a backslash: \%. — JP]

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The at facility submits a command line (or a script) for execution at an arbitrary later time. It has the form:

% options time

This submits scriptfile for execution at a later time. The redirection (<) isn't required on versions that can read directly from a file. By default, at reads the commands from its standard input. So if you don't want to write a script, you can omit the file and type your commands on the terminal, terminated by CTRL-d:

% options time



Command 1

Command 2

...

CTRL-d

The time is most commonly a four-digit number representing a time on a 24-hour clock. For example, 0130 represents 1:30 a.m. and 1400 represents 2 p.m. You can also use abbreviations such as 1am, 130pm, and so on.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most modern versions of at will mail you any output that your commands make. You might think of using the command line below to throw at output into the Unix trash can, /dev/null (Section 43.12):

>& Section 43.5

% sometime...      

               ...wrong

but that won't work because it throws away the output of the at command itself. at just saves your job in a file to be run later by a system program. The commands you want quiet are the commands stored in that file. One way to keep at quiet, if you use a shell like csh , is:

% sometime

at> some command

at> another command

at> ...etc...

at> CTRL-d

Bourne-type shells make it easier:

exec > Section 36.5

$ sometime

at> exec > /dev/null 2>&1

at> some command

at> another command

at> ...etc...

at> CTRL-d

Two notes:

Some versions of at have a -s option that runs your job with the Bourne shell.
Not all versions of at prompt you with at> as I showed above.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

From time to time, you'll submit an at job and realize that there's something wrong with it. How do you get it out of the queue? Two tools help you do this: atq, which reports the jobs that are in the queue, and atrm , which deletes jobs that are already in the queue.

atq is pretty simple; by default, it reports on all jobs that have been queued. Optionally, you can give it a user name as an argument; in this case it reports all the jobs queued by the given user. The report looks like this:

los% atq

 Rank     Execution Date     Owner     Job #   Queue   Job Name

  1st   Oct  9, 1996 22:27   mikel      4637       a   stdin

  2nd   Oct 10, 1996 01:08   mikel      4641       a   stdin

  3rd   Oct 10, 1996 02:34   judy       4663       a   stdin

Note that atq has no objection to telling you about other users' jobs. Although this might seem like a security hole, it's actually useful — see Section 25.8. The jobs are ordered according to their execution date. With the -c option, atq orders jobs according to when they were queued — conceivably a useful feature. (atq -n just prints the number of jobs that are queued; I'm not sure when this would be useful.)

Once you've found out the job number, you can delete it with the command atrm. You can only delete your own jobs, not someone else's:

% atrm 4637

4637: removed

% atrm 4663

4663: permission denied

The command atrm - removes all the jobs you submitted; it's good for cleaning out your queue completely.

On some versions, use at -l to list your jobs (instead of atq) and at -r to delete your jobs (instead of atrm). Other systems may have different commands and options; check your manpage.

Some older BSD-based implementations may not support any of these options. Once you submit a job, you can delete it by finding its filename in the /usr/spool/at directory and emptying the file (Section 15.2). Or the superuser (Section 1.18) can go to the spool directory and delete the file by hand.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

atq and at -l (Section 24.7) are more important than they seem. They give you a way to decide when to run your jobs. I suggest that you check atq before picking a time to run your job. If you don't, the system may have a dozen huge jobs starting at midnight or 1 a.m. They will bring the system to its knees when there's no one around to help out. Here's an example of what can happen, using the BSD-style at commands:

% atq

 Rank      Execution Date    Owner   Job#   Queue  Job Name

 1st    Sep 12, 1996 01:00   mikel   4529     a    trashsys.sh

 2nd    Sep 12, 1996 01:00   johnt   4531     a    flame.sh

 3rd    Sep 12, 1996 01:00   davek   4532     a    stdin

 4th    Sep 12, 1996 01:00   joek    4533     a    troffit

 5th    Sep 13, 1996 02:00   bobr    4534     a    stdin

Four of the five users happened to pick 1 a.m. as their submission time. Therefore, four big jobs will start in the middle of the night. Will your system survive? Will any of these be done in the morning? These are good questions. Instead of submitting your jobs to run at 1 a.m., midnight, or some other integral number, start them at different times, and make them times like 3:48 a.m. If your system administrator notices lots of jobs running at the same times on your system, she might delete some of them and ask you to reschedule.

If your system has personal crontab files (Section 25.2), you won't be able to see other users' cron jobs. The best way to cut system load is to pick strange times like 4:37 a.m. for your cron jobs.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: sleep

The sleep command waits. That's all it does. (GNU versions are usually loaded with features, but the sleep on the CD-ROM [see http://examples.oreilly.com/upt3] doesn't do more than the standard version.) So what good is it?

A quick-and-dirty reminder service when you don't have leave. This will print the message Time to go now.... in 10 minutes (600 seconds):
( ) & Section 43.7, ; Section 28.16
```
% (sleep 600; echo Time to go now....)

&

                  
```
You can't use at (Section 25.5), and you have to run a job later (say, in three hours):
```
% (sleep 10800; someprog) &

                  
```

To watch a program (usually a shell script) that's running in the background and see what processes it runs:

!! Section 30.8

% prog &

[1] 12345

% sleep 5;ps

  PID TT STAT  TIME COMMAND

18305 p4 S     0:01 -csh (csh)

18435 p4 S     0:00 /bin/sh prog

18437 p4 D     0:00 /bin/sort -r temp

18438 p4 R     0:00 ps

% !!;!!;!!;!!;!!

sleep 5; ps; sleep 5; ps; sleep 5; ps; sleep 5; ps; sleep 5; ps

  PID TT STAT  TIME COMMAND

    ...

    ...5 seconds pass...

  PID TT STAT  TIME COMMAND

    ...

When you're running a series of commands that could swamp the computer, to give it time to catch up. For instance, the mail (Section 1.21) program starts background processes to deliver the mail. If you're sending a bunch of form letters, sleep five or ten seconds after each one:
foreach Section 28.9
```
% foreach name (`cat people`)

? formltrprog $name | mail $name

? sleep 10

? end

                  
```
Or, to send print jobs while you're at lunch — but give other people a chance to print between yours:
```
% lp bigfile1;sleep 600;lp bigfile2;sleep 600;lp bigfile3

                  
```

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Whether you are a system administrator or user, the responsiveness of your Unix system is going to be the primary criterion of evaluating your machine. Of course, "responsiveness" is a loaded word. What about your system is responsive? Responsive to whom? How fast does the system need to be to be responsive? There is no one silver bullet that will slay all system latencies, but there are tools that isolate performance bottlenecks — the most important of which you carry on your shoulders.

This chapter deals with issues that affect system performance generally and how you go about finding and attenuating system bottlenecks. Of course, this chapter cannot be a comprehensive guide to how to maximize your system for your needs, since that is far too dependent on the flavors of Unix and the machines on which they run. However, there are principles and programs that are widely available that will help you assess how much more performance you can expect from your hardware.

One of the fundamental illusions in a multiuser, multiprocessing operating system like Unix is that every user and every process is made to think that they are alone on the machine. This is by design. At the kernel level, a program called the scheduler attempts to juggle the needs of each user, providing overall decent performance of:

Keeping interactive sessions responsive
Processing batch jobs promptly
Maximizing CPU utilization
Cranking through as many processes per hour as possible
Preventing any particular process for dominating CPU time

System performance degrades when one of these goals overwhelms the others. These problems are very intuitive: if there are five times the normal number of users logged into your system, chances are that your session will be less responsive than at less busy times.

Performance tuning is a multifaceted problem. At its most basic, performance issues can be looked at as being either global or local problems. Global problems affect the system as a whole and can generally be fixed only by the system administrator. These problems include insufficient RAM or hard drive space, inadequately powerful CPUs, and scanty network bandwidth. The global problems are really the result of a host of local issues, which all involve how each process on the system consumes resources. Often, it is up to the users to fix the bottlenecks in their own processes.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Whether you are a system administrator or user, the responsiveness of your Unix system is going to be the primary criterion of evaluating your machine. Of course, "responsiveness" is a loaded word. What about your system is responsive? Responsive to whom? How fast does the system need to be to be responsive? There is no one silver bullet that will slay all system latencies, but there are tools that isolate performance bottlenecks — the most important of which you carry on your shoulders.

This chapter deals with issues that affect system performance generally and how you go about finding and attenuating system bottlenecks. Of course, this chapter cannot be a comprehensive guide to how to maximize your system for your needs, since that is far too dependent on the flavors of Unix and the machines on which they run. However, there are principles and programs that are widely available that will help you assess how much more performance you can expect from your hardware.

One of the fundamental illusions in a multiuser, multiprocessing operating system like Unix is that every user and every process is made to think that they are alone on the machine. This is by design. At the kernel level, a program called the scheduler attempts to juggle the needs of each user, providing overall decent performance of:

Keeping interactive sessions responsive
Processing batch jobs promptly
Maximizing CPU utilization
Cranking through as many processes per hour as possible
Preventing any particular process for dominating CPU time

System performance degrades when one of these goals overwhelms the others. These problems are very intuitive: if there are five times the normal number of users logged into your system, chances are that your session will be less responsive than at less busy times.

Performance tuning is a multifaceted problem. At its most basic, performance issues can be looked at as being either global or local problems. Global problems affect the system as a whole and can generally be fixed only by the system administrator. These problems include insufficient RAM or hard drive space, inadequately powerful CPUs, and scanty network bandwidth. The global problems are really the result of a host of local issues, which all involve how each process on the system consumes resources. Often, it is up to the users to fix the bottlenecks in their own processes.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Two commands, time and /bin/time, provide simple timings. Their information is highly accurate, because no profiling overhead distorts the program's performance. Neither program provides any analysis on the routine or trace level. They report the total execution time, some other global statistics, and nothing more. You can use them on any program.

time and /bin/time differ primarily in that time is built into many shells, including bash. Therefore, it cannot be used in safely portable Bourne shell scripts or in makefiles. It also cannot be used if you prefer the Bourne shell (sh). /bin/time is an independent executable file and therefore can be used in any situation. To get a simple program timing, enter either time or /bin/time, followed by the command you would normally use to execute the program. For example, to time a program named analyze (that takes two command-line arguments, an input file and an output file), enter the following command:

% time analyze inputdata outputfile

9.0u 6.7s 0:30 18% 23+24k 285+148io 625pf+0w

This result (in the default C shell format) indicates that the program spent 9.0 seconds on behalf of the user (user time), 6.7 seconds on behalf of the system (system time, or time spent executing Unix kernel routines on the user's behalf), and a total of 30 seconds elapsed time. Elapsed time is the wall clock time from the moment you enter the command until it terminates, including time spent waiting for other users, I/O time, etc.

By definition, the elapsed time is greater than your total CPU time and can even be several times larger. You can set programs to be timed automatically (without typing time first) or change the output format by setting shell variables.

The example above shows the CPU time as a percentage of the elapsed time (18 percent). The remaining data reports virtual memory management and I/O statistics. The meaning varies, depending on your shell; check your online

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When your system is sluggish, you will want to see what users are on the system along with the processes they're running. To get a brief snapshot of this information, the tersely named w can show you who is logged in, from where, how long they've been idle, and what programs they're running. For instance, when I run w on my Red Hat box at home, I get this result:

  3:58pm  up 38 days,  4:37,  6 users,  load average: 0.00, 0.07, 0.07

USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT

jjohn    tty2     -                13Feb02  7:03m  1.32s  0.02s  /bin/sh /usr/X

jjohn    pts/1    :0                8:55am  7:02m  0.06s  0.06s  bash

jjohn    pts/3    :0                8:55am  0.00s 51.01s  0.05s  w

jjohn    pts/0    :0                8:55am  7:02m  0.06s  0.06s  bash

jjohn    pts/4    :0                8:55am  2:25m  2:01   0.12s  bash

jjohn    pts/2    mp3.daisypark.ne Tue 4pm  3:41m  0.23s  0.23s  -bash

Originally, I logged in at the console and started X. Most of the sessions are xterminals except for the last, which is an ssh session. The JCPU field accounts for the CPU time used by all the processes at that TTY. The PCPU simply accounts for the process named in the WHAT field. This is a quick and simple command to show you the state of your system, and it relies on no special process accounting from the kernel.

When you're debugging a problem with a program, trying to figure out why your CPU usage bill is so high [in the days when CPU cycles were rented — JJ], or curious what commands someone (including yourself) is running, the lastcomm command on Berkeley-like Unixes can help (if your computer has its process accounting system running, that is). Here's an example that lists the user lesleys:

% date

Mon Sep  4 16:38:13 EDT 2001

% lastcomm lesleys

emacs          lesleys  ttyp1      1.41 secs Wed Sep  4 16:28

cat          X lesleys  ttyp1      0.06 secs Wed Sep  4 16:37

stty           lesleys  ttypa      0.02 secs Wed Sep  4 16:36

tset           lesleys  ttypa      0.12 secs Wed Sep  4 16:36

sed            lesleys  ttypa      0.02 secs Wed Sep  4 16:36

hostname       lesleys  ttypa      0.00 secs Wed Sep  4 16:36

quota          lesleys  ttypa      0.16 secs Wed Sep  4 16:35

   ...

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: uptime

The BSD command uptime, also available under System V Release 4, AIX, and some System V Release 3 implementations, will give you a rough estimate of the system load:

% uptime

3:24pm up 2 days, 2:41, 16 users, load average: 1.90, 1.43, 1.33

uptime reports the current time, the amount of time the system has been up, and three load average figures. The load average is a rough measure of CPU use. These three figures report the average number of processes active during the last minute, the last 5 minutes, and the last 15 minutes. High load averages usually mean that the system is being used heavily and the response time is correspondingly slow. Note that the system's load average does not take into account the priorities and niceness (Section 26.5) of the processes that are running.

What's high? As usual, that depends on your system. Ideally, you'd like a load average under, say, 3, but that's not always possible given what some systems are required to do. Higher load averages are usually more tolerable on machines with more than one processor. Ultimately, "high" means high enough that you don't need uptime to tell you that the system is overloaded — you can tell from its response time.

Furthermore, different systems behave differently under the same load average. For example, on some workstations, running a single CPU-bound background job at the same time as the X Window System (Section 1.22) will bring response to a crawl even though the load average remains quite "low." In the end, load averages are significant only when they differ from whatever is "normal" on your system.

— AF

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The BSD-System V split isn't so obvious in modern Unixes, but the different priority systems still live in various flavors. This article should help you understand the system in whatever version you have.

If you are going to run a CPU-bound ( Section 26.1) process that will monopolize the CPU from other processes, you may reduce the urgency of that more intensive process in the eyes of the process scheduler by using nice before you run the program. For example:

$ nice executable_filename

On most systems, no user can directly change a process's priority (only the scheduler does that), and only the administrator can use nice to make a process more urgent. In practice, nice is rarely used on multiuser systems — the tragedy of the commons — but you may be able to get more processes running simultaneously by judicious use of this program.

If you're not familiar with Unix, you will find its definition of priority confusing — it's the opposite of what you would expect. A process with a high nice number runs at low priority, getting relatively little of the processor's attention; similarly, jobs with a low nice number run at high priority. This is why the nice number is usually called niceness: a job with a lot of niceness is very kind to the other users of your system (i.e., it runs at low priority), while a job with little niceness hogs the CPU. The term "niceness" is awkward, like the priority system itself. Unfortunately, it's the only term that is both accurate (nice numbers are used to compute priorities but are not the priorities themselves) and avoids horrible circumlocutions ("increasing the priority means lowering the priority...").

Many supposedly experienced users claim that nice has virtually no effect. Don't listen to them. As a general rule, reducing the priority of an I/O-bound job (a job that's waiting for I/O a lot of the time) won't change things very much. The system rewards jobs that spend most of their time waiting for I/O by increasing their priority. But reducing the priority of a CPU-bound process can have a significant effect. Compilations, batch typesetting programs (

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It's not a good idea to nice a foreground job (Section 23.3). If the system gets busy, your terminal could "freeze" waiting to get enough CPU time to do something. You may not even be able to kill (Section 24.11) a nice'd job on a very busy system because the CPU may never give the process enough CPU time to recognize the signal waiting for it! And, of course, don't nice an interactive program like a text editor unless you like to wait... :-)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

On Unix systems with BSD-style priority schemes, once a job is running, you can use the renice(8) command to change the job's priority:

% /etc/renice priority-p pid

               

% /etc/renice priority -g pgrp

               

% /etc/renice priority -u uname

where priority is the new nice level (Section 26.5) for the job. It must be a signed integer between -20 and 20. pid is the ID number (Section 24.3) (as shown by ps (Section 24.5)) of the process you want to change. pgrp is the number of a process group ( Section 24.3), as shown by ps -l; this version of the command modifies the priority of all commands in a process group. uname may be a user's name, as shown in /etc/passwd; this form of the command modifies the priority of all jobs submitted by the user.

A nice level of 19 is the "nicest": the process will run only when nothing else on the system wants to. Negative values make a process get a greater percentage of the CPU's time than the default niceness (which is 0). Again, only the superuser can lower the nice number (raise a process' priority). Users can only raise the nice number (lower the priority), and they can modify the priorities of only the jobs they started.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As we've said, the shell is just another program. It's responsible for interpreting the commands you type. There are several commonly used shells, primarily based on two or three major families and a wide variety of other projects:

The Bourne shell (sh) and its derivatives and progeny (including bash, ash, and even the Korn shell ksh)
The C shell (csh) and its progeny (including tcsh)
The Korn shell (ksh) and variants (including pdksh and zsh )
Specialized shells based on languages such as Python, TCL, perl, and so on.
Shells invented to meet specific needs such as restricted command access (rsh), recovery after a system failure (sash), and downloading, installing, and configuring software libraries.

If you can think of a reason to have a specialized shell, someone probably has already written one to meet that need.

Interpreting your commands might seem simple enough, but a lot of things happen between the time you press RETURN and the time the computer actually does what you want. The process of interpretation is very complex: the shell has to break the command into words and expand aliases (Section 29.2), history operators (Section 30.8), and shell and environment variables (Section 35.3, Section 35.9). It also sets up standard input and output streams (Section 43.1) and performs a lot of other tasks. Indeed, if a command looks right but doesn't work right, the cause is probably either one of the following:

File permissions are set incorrectly.
You don't understand how the shell is processing your command line.

I'd say that file permission problems are more common, but it's a close call. File permission problems are usually easy to understand, once you know what to look for, but the rules by which a shell interprets your command line are another thing altogether. Lest I scare you, we'll try to go slow with this material. Although it's difficult, understanding how the shell parses your commands is important to becoming a power user.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As we've said, the shell is just another program. It's responsible for interpreting the commands you type. There are several commonly used shells, primarily based on two or three major families and a wide variety of other projects:

The Bourne shell (sh) and its derivatives and progeny (including bash, ash, and even the Korn shell ksh)
The C shell (csh) and its progeny (including tcsh)
The Korn shell (ksh) and variants (including pdksh and zsh )
Specialized shells based on languages such as Python, TCL, perl, and so on.
Shells invented to meet specific needs such as restricted command access (rsh), recovery after a system failure (sash), and downloading, installing, and configuring software libraries.

If you can think of a reason to have a specialized shell, someone probably has already written one to meet that need.

Interpreting your commands might seem simple enough, but a lot of things happen between the time you press RETURN and the time the computer actually does what you want. The process of interpretation is very complex: the shell has to break the command into words and expand aliases (Section 29.2), history operators (Section 30.8), and shell and environment variables (Section 35.3, Section 35.9). It also sets up standard input and output streams (Section 43.1) and performs a lot of other tasks. Indeed, if a command looks right but doesn't work right, the cause is probably either one of the following:

File permissions are set incorrectly.
You don't understand how the shell is processing your command line.

I'd say that file permission problems are more common, but it's a close call. File permission problems are usually easy to understand, once you know what to look for, but the rules by which a shell interprets your command line are another thing altogether. Lest I scare you, we'll try to go slow with this material. Although it's difficult, understanding how the shell parses your commands is important to becoming a power user.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When the shell executes an external command (Section 1.9), what happens?

Unix programs are executed through a combination of two system calls (low-level requests to the operating system) called fork and exec.

The exec system call tells the kernel to execute another program. However, the kernel replaces the calling program with the new one being called. This doesn't work too well if you want to return to the original program after the new one has done its job.

To get around this problem, programs that want to stick around first copy themselves with the fork system call. Then the copied program execs the new program, terminating itself in the process.

You don't really need to know this little tidbit about what goes on behind the scenes, but it sure helps to know about fork and exec when reading some Unix manuals. Section 24.2 has more information.

—TOR and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A shell is a program that interprets your command lines and runs other programs. Another name for the shell is "command interpreter." This article covers the two major Unix shell families, including discussion about how shells run, search for programs, and read shell script files.

For each command it runs, a shell performs a series of steps. First, if the shell is reading commands from a terminal (interactively), it prints a prompt (such as % or $) and waits for you to type something. Next, the shell reads the command line (like cat -v afile bfile > cfile), interprets it (Section 27.1), and runs the result. When the command finishes running (unless the command is in the background (Section 23.2)), the shell is ready to read another command line.

A shell can read command lines from a terminal or it can read them from a file. When you put command lines into a file, that file is called a shell script (Section 35.1) or shell program. The shell handles the shell script just as it handles the commands you type from a terminal (though the shell uses its non-interactive mode (Section 3.4), which means, basically, that it doesn't print the % or $ prompts, among other things). With this information, you already know how to write simple shell scripts — just put commands in a file and feed them to the shell!

In addition, though, there are a number of programming constructs that make it possible to write shell programs that are much more powerful than just a list of commands.

There are two main shell families in Unix:

The C shell and its derivatives (csh, tcsh) are considered very powerful for situations where you are interactively working on a terminal. csh will read shell scripts and has some useful features for programmers. Unfortunately, it has some quirks that can make shell programming tough.
The Bourne shell (sh) and shells like it are probably used more often for shell programming. (Some newer

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Before getting into the details of command interpretation, I thought I'd give a very simple example of why it's important. Here's an error that occurs all the time. Let's say you have two files, called file1 and file2. You want to create a new version of file1 that has file2 added to the end of it. That's what cat is all about, so you give the command:

% cat file1 file2 > file1          

               ...wrong

This looks like it should work. If you've ever tried it, you know it doesn't; it erases file1, and then dumps file2 into it. Why? The shell (not cat) handles standard input and output:

As the shell is processing the command, it sees that you're redirecting standard output into file1, so it opens the file for writing, destroying the data that's already in it.
Later, after it's finished interpreting the command line, the shell executes cat, passing file1 and file2 as arguments. But file1 is already empty.
cat reads file1 (which is empty) and writes it on standard output (which goes into file1).
cat reads file2 (which also goes into file1). At this point, cat is finished, so it exits.

file1 and file2 are identical, which isn't what you wanted. But it's what you got.

Some versions of cat give you a warning message in this situation (cat: file1:

input file is

output file

). This might lead you to believe that somehow cat was smart and managed to protect you. Sadly, that's not true. By the time cat figures out that an input file and an output file are the same, it's too late: file1 is already gone. This bit of catty cleverness does have a function, though: it prevents commands like the following from creating infinitely long files:

% cat file1 file2 >> file2

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: showargs

When you're experimenting with shell quoting, it's nice to be able to see how arguments on a command line are quoted. Here's a demo of a simple bash script named showargs; you might want to save it in a file and run it yourself (Section 35.1). The script shows how many arguments were passed to it. Then it lists the arguments, one per line, surrounded by >> << to show leading or trailing spaces.

cat Section 12.2, && Section 35.14, $# Section 35.20, path Section 35.7

% cat showargs

#!/bin/bash

test $# -ne 1 && s=s

echo "I got $# argument$s:"

for arg

do echo -E ">>$arg<<"

done

% showargs "Start of path:" $path[1-3] "  that's it!  "

I got 5 arguments:

>>Start of path:<<

>>/u/jpeek/bin<<

>>/bin<<

>>/usr/bin<<

>>  that's it!  <<

The output from your shell may differ from that shown above, which is the result of running showargs in tcsh. bash doesn't have a $path variable, for example. And zsh expects a comma, rather than a hyphen, to separate the range. But as long as the arguments to showargs are quoted properly, you should get the result you're looking for, with a little tweaking, of course!

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Your search path (Section 35.6, Section 35.7) controls what directories — and in what order — the shell searches for external (Section 1.9) commands. You can set a search path that takes effect every time you log in by editing your shell setup file (Section 3.3). You might also want to change the path temporarily. Most shells also keep quick-reference lists of command locations that bypass the search path, so you'll want to know how to manage these.

Changing the path set when you log in is simple: just add the new directory to the appropriate line in your shell's startup files ( Section 3.3). It's not recommended to redefine the path completely, though, as some packages rely on their PATH being set correctly. Usually, it is best simply to add the new directory's absolute path ( Section 31.2) to the end of the existing PATH variable:

PATH=$PATH:$HOME/bin      zsh, sh, ksh, bash

set path=($path ~/bin)    zsh (omit the set), csh, tcsh

If you're configuring the superuser (root) account, be careful about using a path set by the parent process (through $PATH or $path). This path can be used in su shells, giving you part or all the path of the user you su'ed from! Also watch out for a path set by a global setup file like /etc/profile: if it's modified for other users and an insecure version of a system command is added, it could affect the superuser in unexpected ways.

Of course, there's the opposite danger: forgetting to update the superuser's path because you assume that changing the global path will do the job for root too. My advice is to think about it and decide what's best for your system.

For Bourne-type shells, load the updated PATH by typing a command like:

$ .' .profile              

               sh

$ .' .bash_profile         

               bash

For the C shell, type one of these commands, depending on which file you changed:

% source' .cshrc

% source' .tcshrc

% source' .login

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's another example in which command-line parsing is important. Consider this shell alias for counting the number of words in all files:

wc Section 16.6

% alias words "wc -w *" 

               csh, tcsh

$ alias words="wc -w *" 

               ksh, bash

Right away, we can see one effect of command-line parsing. The shell sees the quotation marks and knows not to expand wildcards inside them. Therefore, words is aliased to wc -w *; the * isn't evaluated when you create the alias. (If wildcards were processed before quotes, this wouldn't work.)

Now, think about what happens when you execute the alias. You type:

% words

The shell starts working through its steps and eventually performs alias substitution. When this happens, it converts your command into:

wc -w *

Now, watch carefully. The shell continues working through the process of interpretation (redirection, variable substitution, command substitution) and eventually gets to filename expansion. At this point, the shell sees the * on the command line, expands it, and substitutes the files in the current directory. Seems simple enough. But think: you didn't type this *; the shell put it there when it expanded the wildcard. What would have happened if the shell expanded wildcards before substituting aliases? The * would never have been expanded; by the time the shell put it on the command line, the wildcard expansion stage would be over, and you'd just count the words in a file named * (which probably doesn't exist).

To me, the amazing thing is that all this works — and works well! The workings of the command line are intricate and complex, but the shell almost always does what you want — and without a lot of thought.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you read the previous article (Section 27.7), you saw that, most of the time, the shell evaluates the command line "in the right order." But what about when it doesn't? Here's a situation that the shell can't handle. It's admittedly contrived, but it's not too different from what you might find in a shell program (Section 1.8):

% set b=\$a

% set a=foo

% echo $b

$a

When we use the variable $b, we'd like to get the variable $a, read it, and use its value. But that doesn't happen. Variable substitution happens once, and it isn't recursive. The value of $b is $a, and that's it. You don't go any further.

But there's a loophole. The eval command says, in essence, "Give me another chance. Re-evaluate this line and execute it." Here's what happens if we stick eval before the echo:

% eval echo $b

foo

The shell converts $b into $a; then eval runs through the command-line evaluation process again, converting echo $a into echo foo — which is what we wanted in the first place!

Here's a more realistic example; you see code like this fairly often in Bourne shell scripts:

...

command='grep $grepopts $searchstring $file'

for opt

do

   case "$opt" in

      file) output=' > $ofile' ;;

      read) output=' | more'   ;;

      sort) postproc=' | sort $sortopts';;

   esac

done

...

eval $command $postproc $output

Do you see what's happening? We're constructing a command that will look something like:

grep $grepopts $searchstring $file | sort $sortopts > $ofile

But the entire command is "hidden" in shell variables, including the I/O redirectors and various options. If the eval isn't there, this command will blow up in all sorts of bizarre ways. You'll see messages like

| not

found

, because variable expansion occurs after output redirection. The "nested" variables (like $ofile, which is used inside $output) won't be expanded either, so you'll also see

$ofile not

found

. Putting an eval in front of the command forces the shell to process the line again, guaranteeing that the variables will be expanded properly and that I/O redirection will take place.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

bash, like all shells, performs a series of steps when evaluating a command line. (Sorry, we don't cover all of the Unix shells; we explain bash because it's one of the most common. For other shells, check their manual pages.) This article takes a closer look at how you can control one part of those steps in bash: whether the shell will choose a shell function (Section 29.11), a built-in command ( Section 1.9), or an external command (Section 1.9).

Let's say that you want to write shell functions named cd, pushd, and popd. They will run the shell's built-in cd, pushd, or popd command, respectively, each using the command-line arguments that were passed (via the $@ array reference). Next they execute another shell function named setvars to do some setup in the new directory:

cd( ) {            pushd( ) {         popd( ) {

  cd "$@"           pushd "$@"        popd "$@"

  setvars           setvars           setvars

}                 }                 }

But which cd will bash use when you type cd: the built-in cd or your cd function? (The same question goes for pushd and popd.) Worse, what if the cd <">$@<"> command inside the function makes bash call your cd function again, and that starts an endless loop? Well, that actually will start a loop — and you need to know how to prevent it.

Typing command before the name of a command disables shell function lookup. bash will execute only a built-in command or an external command with that name. So, you could keep the functions from re-executing themselves by defining them this way:

cd( ) {                    pushd( ) {                popd( ) {

    command cd "$@"       command pushd "$@"       command popd "$@"

    setvars               setvars                  setvars

}                         }                        }

In the same way, if you don't want to run your new pushd function for some reason, here's how to use the built-in pushd once:

bash$ command pushd somewhere

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[Section 27.9 shows how to control whether bash uses a built-in command, a shell function, or an external command. The way you do that in the C shell is a little, errr, different. Chris Torek explains why, for example, \rm disables an alias for rm and \cd disables the built-in cd command. He starts with a fairly complex explanation, then gives some practical guidelines. At the end is a "review" that's easy to follow and fun too. — JP]

The C shell first breaks each input line into a word vector. It then matches against aliases. Since \rm does not match rm, any alias is ignored. Eventually the C shell fully applies any quoting (since an alias can include quotes, some of this work must be deferred; since an alias can include multiple words, more word vector work must be done as well; it all gets rather hairy).

The C shell implements quoting by setting the eighth bit (bit 7) of each byte of a quoted character. Since '*'|0x80 [a character ORed with 80 hex, a.k.a. 10000000 binary — JP] is not the same character as '*', this prevents filename expansion, further word breaking, and so on.

Eventually, the shell has a fully "parsed" line. It then compares word[0] [the first word on the command line — JP] against all the built-ins. If there is a match, it runs the corresponding built-in command (and it is up to that command to expand any remaining words; for instance, ls * in a directory containing only the file -l produces a long listing, but

jobs

*

produces a usage message). If not, the shell performs globbing on the current word list, producing a new word list, and then:

Strips the eighth bit of each byte of each word
exec( )s the resulting command.

This means that \cd not only bypasses any alias, but also reaches the built-in scanner as:

'c'|0x80, 'd', '\0'

which does not match the built-in command:

'c', 'd', '\0'

and so does not run the cd builtin. It is later stripped, and the shell looks for an external program called cd.

If you want to avoid alias substitution but not built-in matching, you can replace:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the common questions about Bourne-type shells is why only the second command shown below will redirect both stdout and stderr ( Section 43.1) to a file:

$ cat food 2>&1 >file

cat: can't open food

$ cat food >file 2>&1

$

Although some manual pages don't mention this, the shell processes I/O redirections from left to right:

On the first command line, the shell sees 2>&1 first. That means "make the standard error (file descriptor 2) go to the same place that the standard output (fd1) is going." There's no effect because both fd2 and fd1 are already going to the terminal. Then >file redirects fd1 (stdout) to file. But fd2 (stderr) is still going to the terminal.
On the second command line, the shell sees >file first and redirects stdout to file. Next 2>&1 sends fd2 (stderr) to the same place fd1 is going — that's to the file. And that's what you want.

Section 36.16 has much more about the m >& n operator.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I can't understand why some people see Bourne shell quoting as a scary, mysterious set of many rules. Quoting on Bourne-type shells is simple. (C shell quoting is slightly more complicated. See Section 27.13.)

The overall idea is this: quoting turns off (disables) the special meaning of characters. There are three quoting characters: single quote ('), double quote ("), and backslash (\). Note that a backquote (`) is not a quoting character — it does command substitution ( Section 28.14).

Listed below are the characters that are special to the Bourne shell. You've probably already used some of them. Quoting these characters turns off their special meaning. (Yes, the last three characters are quoting characters. You can quote quoting characters; more on that later.)

# & * ? [ ] ( ) = | ^ ; < > ` $ " ' \

Space, tab, and newline also have special meaning as argument separators. A slash (/) has special meaning to Unix itself, but not to the shell, so quoting doesn't change the meaning of slashes.

Newer shells have a few other special characters. For instance, bash has ! for history substitution (Section 30.8). It's similar to the C shell ! (Section 27.13) except that, in bash, ! loses its special meaning inside single quotes. To find particular differences in your Bourne-type shell, see the quoting section of its manual page. In general, though, the rules below apply to all Bourne-type shells.

Table 27-1 summarizes the rules; you might want to look back at it while you read the examples.

Table 27-1: Bourne shell quoting characters
Quoting character	Explanation
'xxx'

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This article explains quoting in C-type shells by comparing them to Bourne-type shell quoting. If you haven't read Section 27.12 about Bourne shell quoting, please do so now.

As in the Bourne shell, the overall idea of C shell quoting is this: quoting turns off (disables) the special meaning of characters. There are three quoting characters: a single quote ('), a double quote ("), and a backslash (\).

The C shell has a few more special characters in addition to the original Bourne shell:

! { } ~

Table 27-2 summarizes the rules; you might want to look back at it while you read the examples.

Table 27-2: C shell quoting characters
Quoting character	Explanation
'xxx'	Disable all special characters in xxx except `!`.
"xxx"	Disable all special characters in xxx except `$`, ', and `!`.
\x	Disable special meaning of character x. At end of line, a `\` treats the newline character like a space (continues line).

The major differences between C and Bourne shell quoting are the following:

The exclamation point (!) character can be quoted only with a backslash. That's true inside and outside single or double quotes. So you can use history substitution (Section 30.8) inside quotes. For example:
```
% grep intelligent engineering file*.txt

grep: engineering: No such file or directory

% grep '!:1-2' !:3

grep 'intelligent engineering' file*.txt

    ...
```
In the Bourne shell, inside double quotes, a backslash (

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you want to work with files that have spaces or special characters in the filenames, you may have to use quotes. For instance, if you wanted to create a file that has a space in the name, you could use the following:

/dev/null Section 43.12

% cp /dev/null 'a file with spaces in the name'

Normally, the shell uses spaces to determine the end of each argument. Quoting (Section 27.12, Section 27.13) changes that — for example, the cp command above has only two arguments. You can also use a backslash (\) before a special character. The example below will rename a file with a space in the name, changing the space to an underscore ( _ ):

% mv a\ file a_file

Using the same techniques, you can deal with any character in a filename:

% mv '$a' a

At worst, a space in a filename makes the filename difficult to use as an argument. Other characters are dangerous to use in a filename. In particular, using ? and * in a filename is playing with fire. If you want to delete the file a?, you may end up deleting more than the single file.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

C-type shells have two variables that, when set, will help you follow the convoluted trail of variable and metacharacter expansion. This command will echo every command line before shell variables have been evaluated:

set Section 35.9

% set verbose

This command will display each line after the variables and metacharacters have been substituted:

% set echo

If you wish to turn the options off, use unset (Section 35.9) instead of set.

Bourne-type shell syntax is different. To turn on the verbose flag, use:

$ set -v

The command set -x turns on the echo flag. You can also type them together: set -xv.

If your version of Unix understands scripts that start with #!, and nearly all do, here's a convenient way to turn these variables on from the first line of a script:

#!/bin/sh -xv

It is not necessary to modify the program. You can enable variable tracing in Bourne shell scripts by typing the shell name and options on the command line:

$ sh -v script

               

$ sh -x script

Not all Bourne shells let you turn these options off. If yours does (and it probably does), you can do it by using a plus sign instead of a minus sign:

set +xv

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

So far, we've talked about three different kinds of quoting: backslashes (\), single quotes ('), and double quotes ("). The shells support yet one more kind of quoting, called here documents. A here document is useful when you need to read something from standard input, but you don't want to create a file to provide that input; you want to put that input right into your shell script (or type it directly on the command line). To do so, use the << operator, followed by a special word:

sort >file <<EndOfSort

zygote

abacus

EndOfSort

This is very useful because variables (Section 35.9, Section 35.3) are evaluated during this operation. Here is a way to transfer a file using anonymous ftp (Section 1.21) from a shell script:

Go to http://examples.oreilly.com/upt3 for more information on: ftpfile

#!/bin/sh

# Usage:

#     ftpfile machine file

# set -x

SOURCE=$1

FILE=$2

GETHOST="uname -n"

BFILE=`basename $FILE`

ftp -n $SOURCE <<EndFTP

ascii

user anonymous $USER@`$GETHOST`

get $FILE /tmp/$BFILE

EndFTP

As you can see, variable and command substitutions (Section 28.14) are done. If you don't want those to be done, put a backslash in front of the name of the word:

cat >file <<\FunkyStriNG

Notice the funky string. This is done because it is very unlikely that I will want to put that particular combination of characters in any file. You should be warned that the C shell expects the matching word (at the end of the list) to be escaped the same way, i.e., \FunkyStriNG, while the Bourne shell does not. See Section 36.19.

Most Bourne shells also have the <<- operator. The dash (-) at the end tells the shell to strip any TAB characters from the beginning of each line. Use this in shell scripts to indent a section of text without passing those TABs to the command's standard input.

Other shells, notably zsh and later versions of ksh, but in the future possibly also bash, support a method for taking input from a string:

$ tr ... <<< "$xyzzy" | ...

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Before you learn about regular expressions (Section 32.1), you should understand how quoting (Section 27.12) works in Unix.

Regular expressions use metacharacters. The shells also have metacharacters. Metacharacters are simply characters that have a special meaning. The problem occurs when you want to use a regular expression in a shell script. Will the shell do something special with the character? Or will it be passed unchanged to the program? The $ character is a good example of a regular expression metacharacter that is also used by the shell, but whose meaning is different depending upon who interprets it, the shell or other programs. It could be the beginning of a variable name or it could be part of a regular expression (Section 32.2). If you need a regular expression, you must know whether any of the characters of the expression are metacharacters, and must know the right way to quote that character so that it is passed to the program without being modified by the shell.

Table 27-3 is a table of special characters and operators in the shells covered by this book. (Because zsh acts basically like both C-type and Bourne-type shells, its name would show up in every entry. So we don't list it here unless an entry applies only to zsh — or one or two other shells.) The chart also includes several combinations of characters just to be complete. But, to keep things simple, it doesn't include:

Arithmetic operators like +, -, and so on; see the articles on built-in arithmetic for a list.
History substitution like !!, !$, and so on; see Section 30.8 instead.

As in other parts of this book, the sh entries apply to ksh and bash; the csh entries apply to tcsh.

Table 27-3: Special characters and their meanings
Character	Where

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The problem with backslashes is that many different programs use them as quoting characters. As a result, it's difficult to figure out how many backslashes you need in any situation.

Here's an example, taken from System V Release 4. (Notice that I'm using the standard System V version of echo from /bin/echo. SVR4 has four versions of echo!)

% /bin/echo hi \ there

hi  there

% /bin/echo hi \\ there

hi \ there

% /bin/echo hi \ there

hi \ there

In the first case, the shell uses the backslash to quote (Section 27.12) the following space character. The space before the backslash is a word separator. So echo gets two arguments: "hi" and "·there" (without the quotes) — where · is the space character that was quoted by the backslash. As always, echo prints a single space between each argument. The first space you see in the output is echo's argument-separating space, and the second space came along with the second argument (thanks to the backslash).

In the second case, the shell converts \\ to \; the first backslash tells the shell to quote (Section 27.12) (turn off the special meaning of) the second backslash. The echo command gets three arguments, "hi", "\", and "there", and it echoes those arguments with a single space between each. (I've heard claims that, on some systems, this command wouldn't print any backslashes, but I wasn't able to reconstruct that situation.)

In the third case, the shell converts each pair of backslashes into a backslash, and runs the command echo hi \\ there. But this is System V, and System V's echo interprets backslashes as special characters. So when echo sees the remaining two backslashes, it converts them into a single backslash. So you see only a single backslash, even though you typed four. On BSD systems and on Linux, echo doesn't do this; you'd see two backslashes. For that matter, if you're using SVR4's C shell, with its built-in echo command, you'll see the BSD/Linux behavior. You'll also see the BSD/Linux behavior if you're using SVR4's /usr/ucb/echo.

The terminal driver is also capable of "eating" backslashes if they appear before special characters. If a backslash precedes the "erase" character (normally CTRL-h) or the "kill" character (normally CTRL-u), the terminal driver will pass the control character to the shell, rather than interpreting it as an editing character. In the process, it "eats" the backslash. So if you type:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of Unix's best features is the shell's command line. Why? Nearly every modern operating system has a command line; we don't use card readers with obscure job setup cards any more. What makes Unix's special?

The Unix shell command line allows lots of shortcuts. Some of these you'll find in other operating systems; some you won't. In this chapter, we'll introduce a lot of these shortcuts. Among other things, we'll discuss:

How to run commands more than once Section 28.8).
Filename completion ( Section 28.6, Section 28.7), which allows you to type the beginning of a filename and let the shell fill in the rest. (This is finally possible on certain Redmond-born OSes as well, but it usually involves a registry hack or two.)
Command substitution ( Section 28.14), which lets you use the output from one command as arguments to another. (Note that this is different from pipelining.)
Process substitution in bash, and a script named ! for other shells, lets you put the output of a command into a temporary file and give that filename to a process.
The ability to repeat commands with various methods (Section 28.10, Section 28.11).
Handling of command lines that become too long (Section 28.17).

Some fundamental command-line features that we aren't discussing in this chapter, but which are discussed elsewhere, are:

Job control (Section 23.3), which lets you run several commands at the same time.
Aliases (Section 29.2), or abbreviations, for commands. Shell functions (Section 29.11) are similar.
Command-line editing (Section 30.14) and history substitution (Section 30.8). These are two different ways (both useful) to "recall" previous commands.
Quoting (Section 27.12, Section 27.13), the way you "protect" special characters from the Unix shell.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of Unix's best features is the shell's command line. Why? Nearly every modern operating system has a command line; we don't use card readers with obscure job setup cards any more. What makes Unix's special?

The Unix shell command line allows lots of shortcuts. Some of these you'll find in other operating systems; some you won't. In this chapter, we'll introduce a lot of these shortcuts. Among other things, we'll discuss:

How to run commands more than once Section 28.8).
Filename completion ( Section 28.6, Section 28.7), which allows you to type the beginning of a filename and let the shell fill in the rest. (This is finally possible on certain Redmond-born OSes as well, but it usually involves a registry hack or two.)
Command substitution ( Section 28.14), which lets you use the output from one command as arguments to another. (Note that this is different from pipelining.)
Process substitution in bash, and a script named ! for other shells, lets you put the output of a command into a temporary file and give that filename to a process.
The ability to repeat commands with various methods (Section 28.10, Section 28.11).
Handling of command lines that become too long (Section 28.17).

Some fundamental command-line features that we aren't discussing in this chapter, but which are discussed elsewhere, are:

Job control (Section 23.3), which lets you run several commands at the same time.
Aliases (Section 29.2), or abbreviations, for commands. Shell functions (Section 29.11) are similar.
Command-line editing (Section 30.14) and history substitution (Section 30.8). These are two different ways (both useful) to "recall" previous commands.
Quoting (Section 27.12, Section 27.13), the way you "protect" special characters from the Unix shell.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You're logged in from home, running a program and answering a prompt. As you're almost done, modem noise prints xDxD@! on your screen. Where were you? Or you're typing a long command line and a friend interrupts you with write (Section 1.21) to say it's time for lunch. Do you have to press CTRL-u and start typing all over again?

If your system understands the rprnt character (usually set to CTRL-r), you can ask for the command line to be reprinted as it was. In fact, you can use CTRL-r any time you want to know what the system thinks you've typed on the current line — not just when you're interrupted. But this only works in the normal cooked input mode; programs like vi that do their own input processing may treat CTRL-r differently. Here's an example:

% egrep '(10394|29433|49401)' /work/symtower/



Message from alison@ruby on ttyp2 at 12:02 ...

how about lunch?

EOF

CTRL-r

egrep '(10394|29433|49401)' /work/symtower/logs/*

After the interruption, I just pressed CTRL-r. It reprinted the stuff I'd started typing. I finished typing and pressed RETURN to run it.

If you use a shell like the Korn shell that has interactive command editing, you can probably use it to reprint the command line, too. In bash and other commands that use the readline file, though, from vi editing mode, CTRL-r still seems to start an Emacs-style reverse search. So I added this fix to my ~/.inputrc file:

set editing-mode vi



# By default, in vi text-input mode, ^R does Emacs "reverse-i-search".

# In command mode, you can use the vi command ^L to redraw the line.

# Fix it in text-input mode:

"\C-r": redraw-current-line

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The shells' [ ] (square bracket) wildcards will match a range of files. For instance, if you have files named afile, bfile, cfile, and dfile, you can print the first three by typing:

% lpr [a-c]file

Now, let's say that you want to create some more files called efile, ffile, gfile, and hfile. What's wrong with typing the command line below? Try it. Instead of vi, you can use your favorite editor or the touch (Section 14.8) command:

% vi [e-h]file   

               Doesn't make those four files

% ls

afile   bfile   cfile   dfile

Stumped? Take a look at Section 1.13 about wildcard matching.

The answer: wildcards can't match names that don't exist yet. That's especially true with a command like touch ?file (Section 14.8) or

touch

*file

— think how many filenames those wildcards could possibly create!

Section 28.4 explains shell

{

}

operators that solve this problem. And, by the way, if you just created one new file named [e-h]file, simply quote (Section 27.12) its name to remove it:

rm "[e-h]file"

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I've been finding more and more uses for the {} pattern-expansion characters in csh , tcsh, zsh, and bash . They're similar to *, ?, and [ ] (Section 33.2), but they don't match filenames the way that *, ?, and [ ] do. You can give them arbitrary text (not just filenames) to expand — that "expand-anything" ability is what makes them so useful.

Here are some examples to get you thinking:

To fix a typo in a filename (change fixbold5.c fixbold6.c):
```
% mv fixbold{5,6}.c

                  
```
To see what the shell will do with {}, add echo (Section 27.5) before the mv:
```
% echo mv fixbold{5,6}.c

mv fixbold5.c fixbold6.c
```

To copy filename to filename.bak without retyping filename:

% cp filename{,.bak}

To print files from other directory(s) without retyping the whole pathname:

% lpr /usr3/hannah/training/{ed,vi,mail}/lab.{ms,out}

That would give lpr (Section 45.2) all of these files:

/usr3/hannah/training/ed/lab.ms

/usr3/hannah/training/ed/lab.out

/usr3/hannah/training/vi/lab.ms

/usr3/hannah/training/vi/lab.out

/usr3/hannah/training/mail/lab.ms

/usr3/hannah/training/mail/lab.out

...in one fell swoop!

To edit ten new files that don't exist yet:
```
% vi /usr/foo/file{a,b,c,d,e,f,g,h,i,j}

                  
```
That would make /usr/foo/filea, /usr/foo/fileb, ... /usr/foo/filej. Because the files don't exist before the command starts, the wildcard vi /usr/foo/file[a-j] would not work (Section 28.3).
An easy way to step through three-digit numbers 000, 001, ..., 009, 010, 011, ..., 099, 100, 101, ... 299 in the C shell is:
foreach Section 28.9
```
foreach n ({0,1,2}{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9})

   ...Do whatever with the number $n...

end
```
Yes, csh also has built-in arithmetic, but its @ operator can't make numbers with leading zeros. This nice trick shows that the {} operators are good for more than just filenames.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When the C shells, zsh, and bash do history substitutions (Section 30.8) they can also edit the substitution. The C shells and zsh — but not bash — can also edit variable substitutions ( Section 35.9). (bash has a different syntax, which zsh understands, too.) For instance, in the first example below, when !$ contains /a/b/c, adding the "head" operator :h will give just the head of the pathname, /a/b.

For a complete but very terse list of these operators, see the csh manual page. We hope the examples below will help you understand these useful operators.

:h gives the head of a pathname (Section 31.2), as follows:
```
% echo /a/b/c

/a/b/c

% echo !$:h

echo /a/b

/a/b
```
That took off the filename and left the header. This also could be used with C shell variables (Section 35.9) as:
```
% set x = /a/b/c

% echo $x

/a/b/c

% echo $x:h

/a/b
```
:r returns the root of a filename:
```
% echo xyz.c abc.c

xyz.c abc.c

% echo !$:r

echo abc

abc
```
The :r removed the .c from the last argument, leaving the root name. This could also be used in C shell variable names:
```
% set x = abc.c

% echo $x:r

                  
```
:g makes the operation global if you have more than one name. For example:
```
% set x = (a.a b.b c.c)

% echo $x:gr

a b c
```
The :gr operator stripped off all dot (.) suffixes. By the way, this use of g does not work with the history commands.
This is the C shell's answer to the basename (Section 36.13) command.
:e returns the extension (the part of the name after a dot). Using csh variables:
```
% set x=(abc.c)

% echo $x:e

c
```
No luck using that within history, either.
:t gives the tail of a pathname — the actual filename without the path:
```
% echo /a/b/c

/a/b/c

% echo !$:t

c
```
With csh variables:
```
% set x=(/a/b/c)

% echo $x:t

c
```
And with multiple pathnames, you can do it globally with:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you hate typing long filenames, hostnames, command names — or almost anything on a command line — you should know about the shells' "completion" feature.

The basics are pretty simple: just press (in most shells) the TAB key, and the shell should "do the right thing." But how the shell decides what's "right" can be complicated — especially in newer shells, and especially in the latest zsh , which has incredibly customizable completion. As an example, when you press TAB in bash, the shell tries to complete a shell variable if the word begins with $, a username if the word begins with ~, a hostname if the word begins with @, or a command (including aliases and functions). If none of these works, bash finally tries filename completion. As another example, the original Korn shell does only simple filename completion, but the public domain Korn shell has more features.

On more-sophisticated shells, completion is actually a function of the shell's built-in customizable command editor. For instance, in tcsh , the TAB key is bound to (in other words, it runs) the editor's complete-word command. This key binding can be changed. And tcsh, like other recent shells, has plenty of other completion-related editor commands.

bash allows for the customization of the different types of completions, as well; you can define a file containing the hostnames to check (in /etc/hosts format) when the shell is asked to complete a hostname. Just set the environment variable HOSTFILE to the name of the file you want. There are extensive built-in functions in bash, each associated with a key, to allow for extremely flexible management of completions.

As you can see, completion varies shell to shell, so we'll give an overview here. For more details, see your shell's manpage.

Let's look at an example of one type of completion, filename completion. Other types of completion work in generally the same way.

Filename completion

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The shell variable fignore in csh and zsh (FIGNORE in bash and also zsh) lets you tell the shell that you aren't interested in some files when using filename completion (Section 28.6). For example, you may be more likely to refer to C language source files (whose names end with .c) than object files (.o files); you often need to edit your source files, while you may never need to look at your object modules. Set fignore to the suffixes that you want to ignore. For example, to ignore .o files in tcsh and csh, type:

set Section 35.9

% set fignore=(.o)

Once you've done this, file completion will ignore your .o files when you press the TAB key (ESC in csh) — unless a .o file is the only match it can find.

Most likely, there's a whole list of suffixes that you don't care about: .o (object modules), .out (random executables), .gz (gzipped files), ~ (Emacs backup files (Section 19.4)), and so on. Section 1.12 has a list of them. Here's how to set fignore to a list of filenames:

% set fignore=(.o .out .gz \~)   

               ...tcsh, csh, zsh

$ FIGNORE='.o:.out:.gz:~'        

               ...bash, zsh

fignore has no effect when you press CTRL-d to get a listing of the files that match in csh and tcsh. Those shells always give you a complete list of all possible completions.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Let's start with some obvious ways to run commands more than once:

Type !! (Section 30.8) to repeat the previous command line, or repeat a cycle of commands with !-n (Section 30.9)
Press the up-arrow key (Section 30.14) or a vi- or Emacs-style editing command
Copy and paste the command line with your mouse (Section 28.10)

Whether each of those methods will work depends on the shell you're using and whether you have copy-and-paste built into your interface. All of those methods force you to take some action before each command line repeats — pressing the up-arrow key, for instance. That lets you control exactly when each command runs.

The next four articles show automated ways to repeat a command a certain number of times. You can "mix and match" some parts of different articles — the tips on read and sleep, for instance. Each article follows on to the one before, so we suggest glancing through all of them:

In C shells, repeat a single command with the repeat command.
zsh can repeat a series of commands with its repeat loop.
Methods for Bourne-type shells use more-general shell features.
An offbeat method that works with all shells is to output multiple commands using jot.
The shells' for and foreach loops (Section 28.9) can vary the commands they run by picking a string (a word, for instance) from a list of strings.
To repeat a command and display its output in the same place on the screen — so it's easy to spot differences over time — try vis (Section 28.11).

Finally, remember that you aren't stuck with the login shell you chose. If you want a feature that your shell doesn't have, you can use another shell temporarily by typing its name (like csh), running the commands you need, then typing exit to go back to your original shell.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When some people need to repeat a command on several files, the first thing they think of is command line editing (Section 30.14) or — as we show here — history substitution ( Section 30.5):

-v Section 12.4, less Section 12.3

% cat -t -v /usr/fran/report | less

   ...

% ^fran/report^rob/file3

cat -t -v /usr/rob/file3 | less

   ...

% ^3^21

cat -t -v /usr/rob/file21 | less

   ...

%

The second substitution (changing 3 to 21) was quick to do, but the first one was longer. If there are lots of related commands like this, it can be easier to list all the variations at once — then let the shell do the dirty work. To do that, use the shell's foreach loop in C-type shells — or, in Bourne-type shells, use a for loop, shown later in this article. (zsh has both foreach and for loops.) You give the loop a list of the words that will change each time the command line is run. In this example, it's a list of filenames. The loop will step through the words, one by one, storing a word into a shell variable (Section 35.9), then running the command(s). The loop goes on until it has read all the words. For example:

% foreach file (/usr/fran/report /usr/rob/file3 /usr/rob/file21)

? cat -t -v $file | less

? end

                  ...Shell runs cat -t -v /usr/fran/report | less...

                  ...Shell runs cat -t -v /usr/rob/file3 | less...

                  ...Shell runs cat -t -v /usr/rob/file21 | less...

%

The question marks (?) are secondary prompts (Section 28.12); the shell will keep printing them until you type the command end. Then the loop runs.

The list between the parentheses doesn't have to be filenames. Among other things, you can use wildcards (Section 1.13), backquotes (Section 28.14) (command substitution), variables (Section 35.9, Section 35.3), and the handy curly brace ({}) operators

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're using an xterm window (Section 24.20) or another type of terminal emulator with easy copy-and-paste functionality, that might be the easiest way to repeat all or part of a previous command line. Just select the part you want to copy, and paste it at a new prompt, adding any other text before and after pasting. This can be easier than using the shell's editing commands or history operators: what you see is what you get. Figure 28-1 shows copy-and-paste.

Figure 28-1: Copying and pasting a command

You can reuse the copied text over and over, if you want; after copying it once, paste as many times and places as you need to. Also, if you've got multiple pieces of text to copy and paste, try using a scratchpad window or xclipboard (Section 5.19).

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: vis

Sometimes you find yourself repeating the same command over and over again — for example, ps (Section 24.5) to monitor the progress of your background processes, or lpq (Section 45.2) to know when your printout is finished. Instead of typing the same command repeatedly, or even using shell history (Section 30.2) to repeat it, use the vis command. For example:

% vis ps

The vis command takes over your screen and shows the output of the initial ps command. Every 15 seconds, the command is executed again and your screen is updated with the new information. If this delay is too long for you, you can get vis to use a shorter delay using the -d option:

% vis -d 2 ps

The information will now be updated every 2 seconds. Your screen is cleared and you are shown the output of ps. On the top line, vis tells you the command being run, how long your delay is (if not the default), and how many times it has been executed. The Exec: line is incremented every time the command is repeated.

Command:  ps                      Delay:  2            Exec:  1



  PID TT STAT  TIME COMMAND

 2971 p1 S     0:06 -sh (csh)

 6139 p1 S     0:00 vis -d 2 ps

 6145 p1 R     0:00 ps

 3401 q0 IW    0:13 -sh (csh)

 5954 q0 S     0:01 vi ch01

14019 q5 IW    0:02 -sh (csh)

29380 r7 IW    0:00 -bin/csh (csh)

29401 rd IW    0:00 -bin/csh (csh)

vis provides a few other command-line options. The -s option is particularly neat: using -s, any lines that have changed since the last iteration are printed in standout mode.

Note that variations of this command have floated around in the public domain under several different names, such as display, rep, and watch. We found vis to be the most useful.

— LM

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

All shells support multiline commands. In Bourne-type shells, a newline following an open quote (' or "), pipe symbol (|), or backslash (\) will not cause the command to be executed. Instead, you'll get a secondary prompt (from the PS2 shell variable, set to > by default), and you can continue the command on the next line. For example, to send a quick write (Section 1.21) message without making the other user wait for you to type the message, try this:

$ echo "We're leaving in 10 minutes. See you downstairs." |

> write joanne

In the C shells, you can continue a line by typing a backslash (\) before the newline (Section 27.13). In tcsh, you'll see a secondary prompt, a question mark (?), on each continued line. The original csh doesn't prompt in this case.

Obviously, this is a convenience if you're typing a long command line. It is a minor feature and one easily overlooked; however, it makes it much easier to use a program like sed (Section 34.1) from the command line. For example, if you know you chronically make the typos "mvoe" (for "move") and "thier" (for "their"), you might be inspired to type the following command:

nroff -ms Section 3.21, lp Section 45.2

$ sed '

> s/mvoe/move/g

> s/thier/their/g' myfile | nroff -ms | lp

More importantly, the ability to issue multiline commands lets you use the shell's programming features interactively from the command line. In both the Bourne and C shells, multiline programming constructs automatically generate a secondary prompt (> in Bourne shells and ? in C shells) until the construct is completed. This is how our favorite programming constructs for non-programmers, the for and foreach loops (Section 28.9), work. While a simple loop could be saved into a shell script (Section 1.8), it is often even easier to use it interactively.

Here's an example with zsh , which makes secondary prompts that show the names of the construct(s) it's continuing. This

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

<< Section 27.16

The here document operator << (Section 27.16) is often used in shell scripts — but it's also handy at a shell prompt, especially with zsh multiline editing or a throwaway script. But you also can just type it in at a Bourne shell prompt (Section 28.12). (If you use csh or tcsh, you can either use a foreach loop (Section 28.9) or start a subshell (Section 24.4).)

The example below shows a for loop ( Section 28.9) that prints three friendly form letters with the lpr (Section 45.2) command. Each letter has a different person's name and the current date at the top. Each line of the loop body starts with a TAB character, which the <<- operator removes before the printer gets the text:

for person in "Mary Smith" "Doug Jones" "Alison Eddy"

do

   lpr <<- ENDMSG



   `date`



   Dear $person,



   This is your last notice. Buy me pizza tonight or

   else I'll type "rm -r *" when you're not looking.



   This is not a joak.



   Signed,

   The midnight skulker

   ENDMSG

done

The shell reads the standard input until it finds the terminator word, which in this case is ENDMSG. The word ENDMSG has to be on a line all by itself. (Some Bourne shells don't have the <<- operator to remove leading TAB characters. In that case, use << and don't indent the loop body.) The backquotes (Section 28.14) run the date command and output its date; $person is replaced with the person's name set at the top of the loop. The rest of the text is copied as is to the standard input of the lpr command.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A pair of backquotes (``) does command substitution. This is really useful — it lets you use the standard output from one command as arguments to another command.

Here's an example. Assume you want to edit all files in the current directory that contain the word "error." Type this:

-l Section 33.6

$ vi `grep -l error *.c`

3 files to edit

"bar.c" 254 lines, 28338 characters

   ...

$

But why does this work? How did we build the incantation above? First, think about how you'd do this without using any special techniques. You'd use grep to find out which commands contain the word "error"; then you'd use vi to edit this list:

$ grep error *.c

bar.c:  error("input too long");

bar.c:  error("input too long");

baz.c:  error("data formatted incorrectly");

foo.c:  error("can't divide by zero"):

foo.c:  error("insufficient memory"):

$ vi bar.c baz.c foo.c

Is there any way to compress these into one command? Yes, by using command substitution. First, we need to modify our grep command so that it produces only a list of filenames, rather than filenames and text. That's easy; use grep -l:

$ grep -l error *.c

bar.c

baz.c

foo.c

The -l option lists each filename only once, even if many lines in the file match. (This makes me think that grep -l was designed with precisely this application in mind.) Now, we want to edit these files; so we put the grep command inside backquotes, and use it as the argument to vi :

$ vi `grep -l error *.c`

3 files to edit

"bar.c" 254 lines, 28338 characters

   ...

$

You might be wondering about the difference between the "vertical" output from grep and the "horizontal" way that people usually type arguments on a command line. The shell handles this with no problems. Inside backquotes, both a newline and a space are argument separators.

The list you use with command substitution doesn't have to be filenames. Let's see how to send a mail message (Section 1.21) to all the users logged on to the system now. You want a command line like this:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sometimes you need to execute a command with a long list of files for arguments. Here's an easy way to create that list without having to type each filename yourself — put the list in a temporary file:

'...' Section 28.14

% ls > /tmp/mikel

% vi /tmp/mikel

               ...edit out any files you don't want...

% process-the-files 

               `cat /tmp/mikel`

% rm /tmp/mikel

I added the vi step to remind you that you can edit this list; for example, you may want to delete a few files that you don't want to process.

Possible problems: if the list is long enough, you may end up with a command line that's too long for your shell to process. If this happens, use xargs ( Section 28.17). If your system doesn't have xargs, there are other workarounds doesn't that should solve the problem.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When the shell sees a semicolon (;) on a command line, it's treated as a command separator — basically like pressing the ENTER key to execute a command. When would you want to use a semicolon instead of pressing ENTER?

It's nice when you want to execute a series of commands, typing them all at once at a single prompt. You'll see all of them on the same command line and they'll be grouped together in the history list (Section 30.7). This makes it easy to see, later, that you intended this series of commands to be executed one after another. And you can re-execute them all with a simple history command.
As an example, here's a series of commands that puts a listing of the current directory into a temporary file, emails the listing, then overwrites the previous version of the file:
```
$ ll > $tf-1; mail -s backup joe < $tf-1; mv $tf-1 listing

                  
```
I can repeat that same command later by using a history substitution (Section 30.8) like !ll.
It's useful with sleep ( Section 25.9) to run a command after a delay. The next example shows a series of commands in a C shell alias that you might use to print a warning and give the user a chance to abort before the last command (exit, which ends the current shell) is executed. Be sure to read the important note after this example:
```
alias bye 'echo "Type CTRL-c to abort logout"; sleep 10; exit'
```
Note that, in C-type shells and older Bourne-type shells, pressing your interrupt key (Section 24.10) — like CTRL-c — will stop execution of all jobs on the current command line. The alias above works in shells like that. But in some shells, like bash2, interrupting a command in a string of commands separated by semicolons will affect only that single command. So I couldn't rewrite the alias above for bash2 because, if I pressed CTRL-c while the sleep command was executing, that would simply abort sleep — and proceed to run exit, which would log me out immediately!
If you're running a series of commands that take some time to complete, you can type all the commands at once and leave them to run unattended. For example, I have little shell scripts named

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Historically, one of the more annoying things about the design of many UNIX tools was their inability to handle large numbers of arguments. For example, if you wanted to print several hundred files using lpr, you either had to pass them a few at a time, perhaps using wildcards on the command line to split the list up into shorter groups, or call lpr once per file, perhaps using find or a loop. One other method, which is still useful today, involves the use of xargs.

xargs is one of those Unix utilities that seems pretty useless when you first hear about it — but turns into one of the handiest tools you can have.

Go to http://examples.oreilly.com/upt3 for more information on: xargs

If your system doesn't already have xargs, be sure to install it from the web site.

xargs reads a group of arguments from its standard input, then runs a Unix command with that group of arguments. It keeps reading arguments and running the command until it runs out of arguments. The shell's backquotes ( Section 28.14) do the same kind of thing, but they give all the arguments to the command at once. This can give you a Too many arguments error.

Here are some examples:

If you want to print most of the files in a large directory, put the output of ls into a file. Edit the file to leave just the filenames you want printed. Give the file to xargs' standard input:
< Section 43.1
```
% ls > allfiles.tmp

% vi allfiles.tmp

% xargs lpr < allfiles.tmp

                  
```
What did that do? With lines like these in allfiles.tmp:
```
% cat allfiles.tmp

afile

application

   ...

yoyotest

zapme
```
xargs ran one or more lpr commands, each with a group of arguments, until it had read every word in the file:
```
lpr afile application ...

   ...

lpr ... yoyotest zapme
```
This has another advantage for lpr: each print job is fairly short, so you can delete one from the print queue without losing all of them.
The standard output of xargs

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: expect

Expect is a program to control interactive applications such as telnet (Section 1.21) and passwd. These and many other applications interactively prompt and expect a user to enter keystrokes in response. But you can write simple Expect scripts to automate these interactions. Then the Expect program can run the "interactive" program noninteractively. Expect can also be used to automate only parts of a dialogue, since control can be passed from the script to the keyboard and vice versa. This allows a script to do the drudgery and a user to do the fun stuff.

Go to http://examples.oreilly.com/upt3 for more information on: tcl, tk

Expect programs can be written in any language but are almost always written in Tcl. Tcl is an interpreted language that is widely used in many other applications. If you already use a Tcl-based application, you won't have to learn a new language for Expect.

Tcl is a very typical-looking shell-like language. There are commands to set variables (set), control flow (if, while, foreach, etc.), and perform the usual math and string operations. Of course, Unix programs can be called, too.

Expect is integrated on top of Tcl and provides additional commands for interacting with programs. Expect is named after the specific command that waits for output from a program. The expect command is the heart of the Expect program. It describes a list of patterns to watch for. Each pattern is followed by an action; if the pattern is found, the action is executed.

For example, the following fragment is from a script that involves a login. When executed, the script waits for the strings welcome, failed, or busy, and then it evaluates [(executes) — JP] one of the corresponding actions. The action associated with busy shows how multiple commands can be evaluated. The timeout keyword is a special pattern that matches if no other patterns match in a certain amount of time.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In most shells, aliases are an easy way to shorten a long command line or do a short series of commands. Section 29.2 through Section 29.10 cover C shell aliases. Section 29.4 through Section 29.14 cover aliases in bash, pdksh, and zsh.
All except the oldest Bourne-type shells have shell functions (Section 29.11), which are explained in Section 29.11 through Section 29.13. These are a cross between aliases and shell scripts. They're good both for shortening command lines and for running a short or long series of commands.

—JP and SJC

All shells except the original Bourne shell have an "alias" facility that lets you define abbreviations for commands.

The simplest C shell aliases, which are similar to the alias facility in newer Bourne-type shells, are simply a short name for a command and, often, command options or arguments too. The C shell's aliases can get very complicated. Section 29.3 describes how a C shell alias can use arguments from its command line as it's invoked.

As we've said, aliases in Bourne-type shells (bash , zsh, and ksh) are simpler. Section 29.4 covers some of the differences between those shells and the C shells. Still, the ideas for custom C shell commands are useful in any kind of shell, and if you can't write something in a simple Bourne-type alias, you almost certainly can do it in a shell function ( Section 29.11).

You can define aliases from the command line, for use in just your current shell. Any aliases you define can also be placed in your shell setup file (Section 3.3), so they'll be available whenever you're using your shell.

Note that aliases are not passed to subprocesses ( Section 3.3), so putting them in a setup file that's read only by login shells or top-level shells probably isn't what you want. (One exception is an alias for a command that you want to run only in a login shell. For instance, you could define an alias named

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In most shells, aliases are an easy way to shorten a long command line or do a short series of commands. Section 29.2 through Section 29.10 cover C shell aliases. Section 29.4 through Section 29.14 cover aliases in bash, pdksh, and zsh.
All except the oldest Bourne-type shells have shell functions (Section 29.11), which are explained in Section 29.11 through Section 29.13. These are a cross between aliases and shell scripts. They're good both for shortening command lines and for running a short or long series of commands.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

All shells except the original Bourne shell have an "alias" facility that lets you define abbreviations for commands.

The simplest C shell aliases, which are similar to the alias facility in newer Bourne-type shells, are simply a short name for a command and, often, command options or arguments too. The C shell's aliases can get very complicated. Section 29.3 describes how a C shell alias can use arguments from its command line as it's invoked.

As we've said, aliases in Bourne-type shells (bash , zsh, and ksh) are simpler. Section 29.4 covers some of the differences between those shells and the C shells. Still, the ideas for custom C shell commands are useful in any kind of shell, and if you can't write something in a simple Bourne-type alias, you almost certainly can do it in a shell function ( Section 29.11).

You can define aliases from the command line, for use in just your current shell. Any aliases you define can also be placed in your shell setup file (Section 3.3), so they'll be available whenever you're using your shell.

Note that aliases are not passed to subprocesses ( Section 3.3), so putting them in a setup file that's read only by login shells or top-level shells probably isn't what you want. (One exception is an alias for a command that you want to run only in a login shell. For instance, you could define an alias named X that starts your X Window System. If that alias isn't defined in subshells, you'll get a message like

X: command not

found

if you try to start the window system from an existing window.)

A common approach is to create separate files for each shell that store your aliases (such as .bash_aliases for bash or .aliases.csh for the C shell), so that you may source them whenever you like.

Here's one last note that applies to all shells. Anytime you want a list of the aliases currently set, just type alias .

—JP, ML, DG, and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It's convenient for your aliases to use command-line arguments. For example, let's think about an alias named phone:

alias phone 'cat ~/phonelist | grep -i'

After you define that alias, you could type

phone

smith

. The shell would find the phone alias and execute it with the argument (smith) at the end (Section 29.2) this way:

cat ~/phonelist | grep -i smith

Using cat and a pipe that way is inefficient (Section 43.2). It might be more sensible to have an alias that worked like this:

grep -i name ~/phonelist

How do we do this? The C shell's history (Section 30.8) facility lets us use the notation !$ to refer to the last word in the previous command; the notation !* refers to all the arguments of the previous command. Assuming that we only want to look up aliases one at a time, we can use !$ and write our alias like this:

alias phone grep -i \!$ ~/phonelist

When we use the phone command, its final argument will be substituted into the alias. That is, when we type phone bill, the shell executes the command grep -i bill ~/phonelist.

In this example, we needed another kind of quoting. We had to put a backslash before the exclamation point to prevent the shell from replacing !$ with the previous command's last argument. That is, we don't want the shell to expand !$ when we define the alias — that's nonsense. We want the shell to insert the previous argument when we use the alias (in which case, the previous argument is just the argument for the alias itself — clear?).

But why couldn't we just use single quotes or double quotes (Section 27.12)? This isn't the right place for a full explanation, but neither single quotes nor double quotes protect the exclamation point. The backslash does (Section 27.13). If you want to be convinced, experiment with some commands like:

% echo '!!'    

               Print your last command

% echo '\!!'   

               Print !!

The first echo command shows that the shell performs history substitution (i.e., replaces

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A lot of what we said about aliases in Section 29.2 applies to the Korn shell (ksh), zsh, and bash. This article, along with Section 29.5 and Section 29.6, have an overview of what's different.

One thing that's different from C shells is the syntax of the alias command, which is:

$ alias 

               name

               =

               definition

That is, you need an equal sign (no spaces) between the name and the definition. A good guideline is to use single quotes (') around the definition unless you're doing something specialized and you understand how quoting (Section 27.12) works in aliases.

You can't put arguments inside an alias as the C shell's \! operator ( Section 29.3) does. To do that, use a shell function ( Section 29.11).

As in the C shells, unalias removes an alias. To remove all aliases, use unalias -a in ksh and bash or unhash -a in zsh. alias with no arguments lists aliases that are currently defined.

bash aliases are pretty basic; this section covers them. Korn shell and zsh aliases do more.

—JP and SC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

pdksh (the public domain ksh) has three types of aliases. First is the regular command alias covered in Section 29.4.

Tracked aliases keep track of the locations of external (Section 1.9) executables. The shell has a default list of commands to track (see the ksh manpage). The first time ksh searches the PATH for an executable command that's marked as a tracked alias, it saves the full path of that command. This saves the shell the trouble of performing the path search each time a command is invoked. The tracked aliases aren't reset unless one becomes invalid or you change the PATH. The command alias -t lists and creates tracked aliases. Here's an example with a newly invoked Korn shell:

$ alias -t

$ cat somefile > somewhere

$ alias -t

cat=/bin/cat

$ alias -t less

$ alias -t

cat=/bin/cat

less=/usr/bin/less

At first, there are no tracked aliases. But the cat command is marked for tracking; as soon as I use it, the shell saves its location, as the next

alias

-t

shows. Next, I add a tracked alias for less (Section 12.3) (which isn't one of the default commands to track). The Korn shell won't track a command unless it's one of the defaults or you mark it for tracking.

The third kind of alias, directory aliases, set with alias -d, let you use a tilde abbreviation like ~ dir for any directory.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

zsh has the regular command alias covered in Section 29.4. zsh is compatible with the C shell in many ways, but it doesn't accept csh alias syntax without an equal sign (= ) between the name and value. That's probably because, as in other Bourne-type shells, zsh allows you to set multiple aliases with one command, like this:

zsh$ alias ri='rm -i' mi='mv -i'  

               ...and so on

In zsh, alias -g defines a zsh global alias: a word that's expanded anywhere (as long as it isn't quoted). These are like a shell variable (Section 35.9) that doesn't need a dollar sign ($) to be expanded. Maybe you have a log file you read and edit often. You could make a global alias named log:

zsh$ alias -g log=/work/beta/p2/worklog



zsh$ less log

zsh$ cp log logtemp

Global aliases are expanded only when they stand alone and aren't quoted. So if there's a global alias dir for a directory, you cannot use emacs dir/file to refer to a file in that directory. Also, if you defined the global alias fserv for the hostname fserv.bkk.ac.uk, you could type

telnet

fserv

— but if you type mail ed@fserv, the shell wouldn't expand it into a hostname. Named directories and shell variables work better in cases like those.

alias -m lists aliases that match a wildcard-type pattern; alias -m 'hi*' shows all alias names that start with hi (like hi, hist, and so on). This matches regular command aliases as well as global aliases. You can use -m with unalias, too, to remove all aliases matching a pattern.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Aliases are a powerful concept in csh. Another great capability is shell scripts (Section 1.8). Each has its strengths. An alias is just right for common sequences of commands, calling a command by a different name, and so on. Scripts are great for more flexible processing and batch processing. There are limitations to both, and I will show a way around them.

The limitation to aliases is that you are working pretty much with one command line. Consider this example, which manages various stages of changing directories, updating the prompt, and so forth:

alias pp 'set o2=$cwd; popd; set old=$o2; dir_number; record_dir pp; \\

  prompt_set; set cd_attempt=(\!*); if ($#cd_attempt > 0) cd $cd_attempt'

Now this works fine for me, and it served me well for a few years and thousands of invocations, but it's at the point where I start thinking that a script is more suited to the job. This brings me to the limitation of scripts.

Shell scripts are great for accomplishing some task that might change a file, start a program, etc. They are limited by the fact that any changes they make to shell or environment variables are not visible (Section 24.3) to the parent shell that started them. In other words, you can write some really cool script that will change directories for you if you don't touch the keyboard for five seconds, but once the script exits, you are still in the same place you started.

The answer is to combine the best of both worlds. Consider this:

alias pp 'set cd_attempt=(\!*); source ~/bin/pp_csh'

We set up a variable and source a script. The concept is this: put your command-line arguments into a variable and then source ( Section 35.29) a script to accomplish something. The difference here is that because you are not starting a subshell (Section 24.4) for the script, it can do everything an alias can and more. This is much like Bourne shell functions (Section 29.11).

Some hints on using this technique:

Naming: I like to name the script that is doing all of the work after the alias, with

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 27.9 has similar information for bash.

Here's a situation that came up on the Net a while ago. Someone wanted an exit (Section 24.4) alias that would run a ~/.exit file (Section 31.13) before leaving the shell. The obvious solution is:

alias exit "source ~/.exit; exit"

This doesn't work; when you use the exit alias, the C shell thinks that the alias is trying to execute itself. Recursive aliases aren't allowed on many shells, so the C shell prints an error message (Alias loop) and gives up.

There are many ways to break the loop. Here's the best (in my opinion):

alias exit 'source ~/.exit; ""exit'

Section 27.10 has the hairy details of what works and why. To summarize, if you need to use the alias's name within a C shell alias, you can use:

""name: Where name is the name of a built-in (Section 1.9) command or any "regular" command.
\name: Where name is the name of any "regular" command, but not a built-in command.

Tempting as this all may sound (and I have to admit, if it didn't sound a bit tempting, I wouldn't be writing this article), I can't really recommend the practice of "redefining" commands with aliases. You should leave the original commands as they are. The original author could have avoided all these problems by calling his alias quit rather than exit.

If you redefine commands with aliases and then use another account where your alias isn't defined, it's easy for things to go wrong. That's especially true for commands that do something permanent — overwriting or removing files, for example. It also can cause problems if you let someone type a command on your account and the person isn't expecting an aliased version.

Let me give one more example to show you what problems you can have. Let's say you've aliased the exit command to source a .exit file before quitting. Fair enough. But now, let's say that you're not in your login shell, that you've set ignoreeof, and that, for no apparent reason, your .exit file disappears (maybe it develops a bad block, so the system can't read it; such things happen).

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The C shell's brain damage keeps you from using an if with an else in an alias. You have to use a sourceable script (Section 29.7). Or that's what I thought until I saw an article by Lloyd Zusman on comp.unix.questions in December 1987. He'd saved an earlier posting on that group (but without its author's name) that showed how. The trick: use enough backslashes (\) and the eval (Section 27.8) command.

As an example, here's an alias named C for compiling C programs. It needs the executable filename (like

C

prog

), not the source filename (like

C

prog.c

). If you type a filename ending in .c, it complains and quits. Else, it does the following:

Renames any old prog file to prog.old.
Prints the message prog SENT TO cc.
Compiles prog.c.
And — if there's a prog file (if the compile succeeded) — runs chmod 311 prog to protect the file from accidental reading with a command like cat * or more *.

Your alias doesn't need to be as complicated. But this one shows some tricks, such as putting an if inside the if, that you might want to use. Watch your quoting — remember that the shell strips off one level of quoting when you set the alias (Section 29.3) and another during the first pass of the eval. Follow this example and you'll probably be fine:

Go to http://examples.oreilly.com/upt3 for more information on: if-else-alias.cs

# COMPILE AND chmod C PROGRAMS; DON'T USE .c ON END OF FILENAME.

alias C 'eval "if (\!* =~ *.c) then \\

   echo "C quitting: no .c on end of \!* please." \\

else \\

   if (-e \!*) mv \!* \!*.old \\

   echo \!*.c SENT TO cc \\

   cc -s \!*.c -o \!* \\

   if (-e \!*) chmod 311 \!* \\

endif"'

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Getting quoting right in C shell aliases can be a real problem. Dan Bernstein wrote two aliases called makealias and quote that take care of this for you.

For example, here I use makealias to avoid having to quote ! and *:

% makealias mycat

cat `ls | sed '1,/!*/d'` | less

CTRL-d

alias mycat 'cat `ls | sed '\''1,/\!*/d'\''` | less'

I typed the makealias mycat command and the line starting with cat, then pressed CTRL-d and got back an alias definition with all the quoting done correctly.

The properly quoted alias definition is sent to the standard output. That line is what you would use to define the alias.

Here are the quote and makealias aliases themselves:

Go to http://examples.oreilly.com/upt3 for more information on: makealias.csh

alias quote     "/bin/sed -e 's/\\!/\\\\\!/g' \\

   -e 's/'\\\''/'\\\'\\\\\\\'\\\''/g' \\

   -e 's/^/'\''/' -e 's/"\$"/'\''/'"

alias makealias "quote | /bin/sed 's/^/alias \!:1 /' \!:2*"

Pretty gross, but they do the job. On Darwin, as on many BSD-derived systems, sed is in /usr/bin, not /bin. You may wish simply to use the command name without the explicit path, or use the explicit (but correct) path. On Linux, the script above does not work with tcsh, which handles multi-line aliases anyway.

—JIK and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most shells have aliases ( Section 29.2). Almost all Bourne-type shells have functions, which are like aliases, but richer and more flexible. Here are four examples.

Let's start with two aliases from Section 29.2, changed into shell functions: The la function includes "hidden" files in ls listings. The lf function labels the names as directories, executable files, and so on.

function la ( ) { ls -a "$@"; }

function lf ( ) { ls -F "$@"; }

The spaces and the semicolon (;) are important. You don't need them on some shells, but writing functions this way (or in the multiline format in later examples) is more portable. The function keyword is not needed in the original Bourne shell but is required in later versions of bash. The "$@" ( Section 35.20) is replaced by any arguments (other options, or directory and filenames) you pass to the function:

$ la -l somedir            

                  ...runs ls -a -l somedir

Go to http://examples.oreilly.com/upt3 for more information on: mx.sh

The mx function uses dig to look up the DNS MX (mail exchanger) record for a host, then sed (Section 34.1) to pull out the "ANSWER SECTION", which has the hostname or hostnames:

for Section 35.21

function mx( ) {

# Look up mail exchanger for host(s)

for host

do

    echo "==== $host ===="

    dig "$host" mx in |

    sed -n '/^;; ANSWER SECTION:/,/^$/{

            s/^[^;].* //p

    }'

done

}

mx takes one or more hostname arguments; it runs dig and sed on each hostname. For example, the mail exchangers for oreilly.com are smtp2.oreilly.com and smtp.oreilly.com. The mail exchanger for hesketh.com is mail.hesketh.com:

$ mx oreilly.com hesketh.com

==== oreilly.com ====

smtp2.oreilly.com.

smtp.oreilly.com.

==== hesketh.com ====

mail.hesketh.com.

This example shows how to write a function with more than one line. In that style, with the ending curly brace on its own line, you don't need a semicolon after the last command. (The curly braces in the middle of the function are inside quotes, so they're passed to

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 27.11 introduces shell functions for all Bourne-type shells. This article covers details of functions in specific shells.

Read-only functions

A bash and ksh function can be made read-only. In ksh, that means the function can't be changed. In bash, it can't be changed or removed. To make a function read-only, use the ksh command typeset -r funcname or use read-only -f funcname in bash, where funcname is the name of the function.

A system administrator might want to set read-only functions from a system-wide setup file (Section 3.3) like /etc/profile. bash users can't unset read-only functions, though. So once a function foo has been defined, how can you define your own foo? As Section 27.9 explains, you can type command foo to use a command named foo from your search path. Or define an alias named foo; aliases are used before functions. Finally, if you'd like to redefine the function, make an alias with the same name, then make the alias invoke a function with a (usually similar) name. For instance, to override a read-only function named foo:

alias foo=_foo

function _foo( ) {

    ...your foo function...

}

Changing function environment

If a function uses an environment variable — like VISUAL or EDITOR (Section 35.5), your standard text editor — you can set the value temporarily while the function executes. The syntax is the same for functions, but only in bash and zsh . For instance, if you usually use vi, but you want to use emacs as you run the work function ( Section 29.11):

$ VISUAL=emacs work todo

Resetting zsh options

There are lots of zsh options. You may want to set some of them temporarily during a shell function without needing to reset them to their previous values when the function returns to the calling shell. To make that happen, set the LOCAL_OPTIONS option (run setopt local_options) in the function body.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One easy way to define shell functions that you'll have every time you start a new shell is by defining them in your shell setup files (Section 3.3). Here are two other ways.

In bash , you can export functions to other bash subshells (Section 24.4). (The original Korn shell, but not the public-domain version, supposedly does this too, but I haven't had much luck with it.) Just use the command typeset -fx funcname, where funcname is the name of the function.

How does this work? It stores the function in an environment variable (Section 35.3) whose value starts with

(

)

. You can see this with printenv or env (Section 35.3). For example, let's define a simple function named dir, export it, start a subshell, run the function, and look for it in the environment:

bash$ function dir( ) { ls -F "$@"; }

bash$ typeset -fx dir         

                  ...export the function

bash$ bash                    

                  ...start subshell

bash$ dir                     

                  ...the function still works

,ptbk.last            ch14.sgm    ch36.ps.gz          fmt/

,xrefs.list           ch15.ps.gz  ch36.sgm            gmatlogs/

bash$ printenv

                  ...lots of environment variables...

dir=( ) {  ls -F "$@"

}

Both ksh and zsh will automatically search for functions in the PATH variable ( Section 35.6). So you can put a function in a file with the same name as the function (for instance, put the function foo in a file named foo), and make the file executable (with chmod +x foo (Section 35.1)), and then the shell can find the function.

I don't like to use PATH for function-searching, though. One reason is that PATH is passed to all Unix processes — but if the process isn't a shell and it tries to execute a function file, it'll probably fail in an ugly way. Also, making a file executable if you don't tell the kernel how to execute it seems to me a recipe for trouble. A better way to help the shell find functions is to set a function search path in the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Until System V Release 2 (circa 1984), the Bourne shell had no way for users to set up their own built-in commands. If you have a Bourne shell with no functions (Section 29.11) or aliases (Section 29.2) and haven't yet turned the host machine into a wet bar, CD/DVD storage case, or some other pragmatic but fun use for a 30-year-old computer, you can do a lot of the same things with shell variables and the eval (Section 27.8) command.

Let's look at an example. First, here's a shell function named cps (copy safely). If the destination file exists and isn't empty, the function prints an error message instead of copying:

test Section 35.26

cps( )

{

   if test ! -s "$2"

   then cp "$1" "$2"

   else echo "cps: cannot copy $1: $2 exists"

   fi

}

If you use the same cps twice, the first time you'll make bfile. The second time you try, you see the error:

$ cps afile bfile

   ...

$ cps afile bfile

cps: cannot copy afile: bfile exists

Here's the same cps — stored in a shell variable instead of a function:

cps='

if test ! -s "$2"

then cp "$1" "$2"

else echo "cps: cannot copy $1: $2 exists"

fi

'

Because this fake function uses shell parameters, you have to add an extra step: setting the parameters. Simpler functions are easier to use:

set Section 35.25

$ set afile bfile

$ eval "$cps"

   ...

$ eval "$cps"

cps: cannot copy afile: bfile exists

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It has been said that "the only thing we learn from history is that people don't learn from history."

Fortunately, the original maxim that "history repeats itself" is more appropriate to Unix.

Most shells include a powerful history mechanism that lets you recall and repeat past commands, potentially editing them before execution. This can be a godsend, especially when typing a long or complex command.

All that is needed to set C shell history in motion is a command like this in your .cshrc (or .tcshrc) file, where n is the number of past commands that you want to save:

set history=n

In ksh and bash, the variable is HISTSIZE, and it's already set for you; the default values are 128 and 500, respectively.

The history command (Section 30.7) lists the saved commands, each with an identifying number. (It's also possible to configure the shells to print the history number of each command as part of your prompt (Section 4.3).)

In tcsh, csh, and bash, you can repeat a past command by typing its number (or its name) preceded by an exclamation point (!). You can also select only parts of the command to be repeated and use various editing operators to modify it. Section 30.8 and Section 28.5 give quick tutorial summaries of some of the wonderful things you can do. Most of the rest of the chapter gives a miscellany of tips for using and abusing the shells' history mechanism.

Most shells — except the original Bourne and C shells — also have interactive command-line editing (Section 30.14). Interactive editing might seem to be better than typing !vi or lpr !$. If you learn both systems, though, you'll find plenty of cases where the ! system is faster and more useful than interactive editing.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It has been said that "the only thing we learn from history is that people don't learn from history."

Fortunately, the original maxim that "history repeats itself" is more appropriate to Unix.

Most shells include a powerful history mechanism that lets you recall and repeat past commands, potentially editing them before execution. This can be a godsend, especially when typing a long or complex command.

All that is needed to set C shell history in motion is a command like this in your .cshrc (or .tcshrc) file, where n is the number of past commands that you want to save:

set history=n

In ksh and bash, the variable is HISTSIZE, and it's already set for you; the default values are 128 and 500, respectively.

The history command (Section 30.7) lists the saved commands, each with an identifying number. (It's also possible to configure the shells to print the history number of each command as part of your prompt (Section 4.3).)

In tcsh, csh, and bash, you can repeat a past command by typing its number (or its name) preceded by an exclamation point (!). You can also select only parts of the command to be repeated and use various editing operators to modify it. Section 30.8 and Section 28.5 give quick tutorial summaries of some of the wonderful things you can do. Most of the rest of the chapter gives a miscellany of tips for using and abusing the shells' history mechanism.

Most shells — except the original Bourne and C shells — also have interactive command-line editing (Section 30.14). Interactive editing might seem to be better than typing !vi or lpr !$. If you learn both systems, though, you'll find plenty of cases where the ! system is faster and more useful than interactive editing.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The C shell and bash can save copies of the previous command lines you type. Later, you can ask for a copy of some or all of a previous command line. That can save time and retyping.

This feature is called history substitution , and it's done when you type a string that starts with an exclamation point (! command). You can think of it like variable substitution ($varname) (Section 35.9) or command substitution ('command') (Section 28.14): the shell replaces what you type (like !$) with something else (in this case, part or all of a previous command line).

Section 30.1 is an introduction to shell history. These articles show lots of ways to use history substitution:

We start with favorite uses from several contributors — Section 30.3, Section 30.4, Section 30.5, and Section 30.6.
Section 30.8 starts with a quick introduction, then covers the full range of history substitutions with a series of examples that show the different kinds of things you can do with history.
(Back in Section 28.5 are examples of csh/tcsh and bash operators such as :r. Many of these can be used to edit history substitutions.)
See an easy way to repeat a set of csh/tcsh or bash commands in Section 30.9.
Each shell saves its own history. To pass a shell's history to another shell, see Section 30.12 and Section 30.13.
You don't have to use an exclamation point (!) for history. Section 30.15 shows how to use some other character.
The Korn shell does history in a different way. Section 30.14 introduces part of that: command-line editing in ksh and bash.

One last note: putting the history number in your prompt ( Section 4.3) makes it easy to reuse commands that haven't scrolled off your screen.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I use !$ so much that it's almost a single character to me. It means "take the last thing on the previous command line." Since most Unix commands have the filename last, you often need to type filenames only once, and then you can use !$ in subsequent lines. Here are some examples of where it comes in handy:

I get a lot of tar archives (Section 39.2). To extract and edit a file from them, I first make a backup for easy comparison after editing:
```
% tar xzf prog.1.05.tar.gz foo.c

% cp -i !$ !$.orig

cp -i foo.c foo.c.orig
```
The same trick is also good when you've edited a file with vi and then want to check its spelling:
```
% vi fred.letter.txt

% ispell !$

                  
```
You often want to move a file to another directory and then cd to that directory. The !$ sequence can also be used to refer to a directory:
```
% mv grmacs.tar /usr/lib/tmac

% cd !$

cd /usr/lib/tmac
```

— AN

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I use !$ (Section 30.3) a lot, but my favorite history substitution is !:n *, where n is a number from 0 to 9. It means "take arguments n through the last argument on the previous command line." Since I tend to use more than one argument with Unix commands, this lets me type the arguments (usually filenames) only once. For example, to use RCS (Section 39.5) and make an edit to article files named Section 35.5 and Section 29.2 for this book, I did:

% co -l 1171.sgm 6830.sgm 2340.sgm

RCS/1171.sgm,v  ->  1171.sgm

   ...

RCS/2340.sgm,v  ->  2340.sgm

revision 1.8 (locked)

done

% vi !:2*

vi 1171.sgm 6830.sgm 2340.sgm

3 files to edit

   ...

% ci -m"Changed TERM xref." !*

ci -m"Changed TERM xref." 1171.sgm 6830.sgm 2340.sgm

   ...

In the first command line (co), I typed the filenames as arguments 2, 3, and 4. In the second command line (vi), I used !:2*; which grabbed arguments 2 through the last (in this case, argument 4) from the first command line. The result was a second command line that had those three filenames as its arguments 1, 2, and 3. So, in the third command line (ci), I used !* to pick arguments 1 through the last from the previous (second) command line. (!* is shorthand for !:1*.)

You can also grab arguments from previous command lines. For example, !em:2* grabs the second through last arguments on the previous emacs command line (command line starting with "em"). There are lots more of these in Section 30.8.

If these examples look complicated, they won't be for long. Just learn to count to the first argument you want to grab. It took me years to start using these substitutions — but they've saved me so much typing that I'm sorry I didn't get started earlier!

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Well, maybe it's not my favorite, but it's probably the history substitution I use most often. It's especially handy if you have fumble-fingers on a strange keyboard:

% cat myflie

cat: myflie: No such file or directory

% ^li^il

cat myfile

Obviously, this doesn't save much typing for a short command, but it can sure be handy with a long one. I also use ^^ with :p ( Section 30.11) to recall an earlier command so I can change it. For example:

% !m:p

more gobbledygook.c

% ^k^k2

more gobbledygook2.c

The point is sometimes not to save typing, but to save the effort of remembering, such as, I want to print the file I looked at earlier, but don't remember the exact name.

[My keyboard can repeat characters when I'm not expecting it. I use a single ^ to delete extra characters. For example:

% lpr sources/aproggg.c

lpr: sources/aproggg.c: no such file or directory

% ^gg

lpr sources/aprog.c

You could type ^gg^, but the second caret isn't required. With a single caret, you don't type a replacement string — just the string to delete. — JP]

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We all know about using ls before a wildcarded rm to make sure that we're only deleting what we want. But that doesn't really solve the problem: you can type ls a* and then mistakenly type

rm

s*

with bad consequences — it's just a minor slip of your finger. But what will always work is:

% ls a*

a1 a2 a3

% rm !$

(ls -d a* (Section 8.5) will make less output if any subdirectory names match the wildcard.)

Using the history mechanism to grab the previous command's arguments is a good way to prevent mistakes.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most of the history examples we've shown use the first few letters in a command's name: !em to repeat the previous Emacs command, for example. But you also can recall previous commands by their numbered position in the history list. That's useful when you have several command lines that start with the same command. It's also more useful than interactive command-line editing (Section 30.14) when you'd like to see a lot of previous commands at once and then choose one of them by number.

To list previous commands, use the history command. For instance, in bash and the C shells, history 20 shows your last 20 commands. In zsh and the Korn shell, use a hyphen before the number: history -20 (also see the discussion of fc, later in this article). Here's an example:

$ history 8

 15  show last +upt/authors

 16  vi ../todo

 17  co -l 0444.sgm

 18  vi 0444.sgm

 19  ci -u 0444.sgm

 20  rcsdiff -u3.4 0444.sgm > /tmp/0444-diff.txt

 21  scp /tmp/0444-diff.txt webhost:adir/11.03-diff.txt

 22  getmail;ndown

$ rm !20:$

rm /tmp/0444-diff.txt

$ !16

vi ../todo

The number at the start of each line is the history number. So, to remove the temporary file I created in command 20 (the name of which I'd already forgotten!), I can use !20:$ (Section 30.8) to pass that filename as an argument to rm. And to repeat command 16 (vi ../todo), I can type !16.

This sort of thing is often faster than using arrow keys and editor commands to recall and edit previous commands. It lets me see several commands all at once, which makes it easier to spot the one(s) I want and to remember what I was doing as I worked. I use it so often that I've got a set of aliases that list bigger and bigger chunks of previous commands and an alias that searches history, giving me a chunk of matching command lines. Here they are in C shell syntax:

less Section 12.3, \!* Section 29.3

alias h     history 5     # show last five lines

alias hi    history 10    # show last ten lines

alias his   history 20    # show last 20 lines

alias hist  'history 40 | less'        # show last 40; pipe to 'less'

alias histo 'history 70 | less'        # show last 70; pipe to 'less'

alias H     'history -r | fgrep "\!*"' # find something in history

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Although most of the examples here use echo to demonstrate clearly just what is going on, you'll normally use history with other Unix commands.

The exclamation point (!) is the default (Section 30.15) history substitution character. This allows you to recall previously entered commands and re-execute them without retyping. To use the ! in a command line, you have several choices. Some of the following examples are more of a headache than they may be worth, but they are also used to select arguments from the command line in aliases (Section 29.3):

!! repeats the last command.
!: repeats the last command. This form is used if you want to add a modifier (Section 28.5) like the following:
```
% echo xy

xy

% !:s/xy/yx

echo yx

yx
```
The second ! was left out.
!so repeats the last command that starts with so.
!?fn? repeats the last command that has fn anywhere in it. The string could be found in an argument or in the command name. This is opposed to !fn, in which !fn must be in a command name. (The last ? need not be there. Thus !?fn means the same thing.)
!34executes command number 34. You can find the appropriate history number when you list your history using the history command, or by putting the history number in your prompt (Section 4.3).
!! & adds an ampersand (&) to the end of the last command, which executes it and places it into the background. You can add anything to the end of a previous command. For example:
```
% cat -v foo

       ...

% !! | more

cat -v foo | more

       ...
```
In this case the shell will repeat the command to be executed and run it, adding the pipe through the more pager. Another common usage is:
```
% cat -v foo

       ...

% !! > out

cat -v foo > out
```
which returns the command but redirects the output into a file.
!:0selects only the command name, rather than the entire command line.
```
% /usr/bin/grep Ah fn1
```

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The !! history substitution gives a copy of the previous command. Most people use it to re-execute the previous command line. Sometimes I want to repeat a cycle of two commands, one after the other. To do that, I just type !-2 (second-previous command) over and over:

% vi plot

   ...

% vtroff -w plot

   ...

% !-2

vi plot

   ...

% !-2

vtroff -w plot

   ...

You can cycle through three commands with !-3, four commands with !-4, and so on. The best part is that if you can count, you never have to remember what to do next. :-)

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[There are times when history is not the best way to repeat commands. Here, Jerry gives an example where a few well-chosen aliases can make a sequence of commands, all run on the same file, even easier to execute. — TOR]

While I was writing the articles for this book, I needed to look through a set of files, one by one, and run certain commands on some of those files. I couldn't know which files would need which commands, or in what order. So I typed a few temporary aliases on the C shell command line. (I could have used shell functions (Section 29.11) on sh-like shells.) Most of these aliases run RCS (Section 39.5) commands, but they could run any Unix command (compilers, debuggers, printers, and so on).

% alias h 'set f="\!*";co -p -q "$f" | grep NOTE'

% alias o 'co -l "$f"'

% alias v 'vi "$f"'

% alias i 'ci -m"Fixed NOTE." "$f"'

The h alias stores the filename in a shell variable (Section 35.9). Then it runs a command on that file. What's nice is that, after I use h once, I don't need to type the filename again. Other aliases get the filename from $f:

% h ch01_summary

NOTE: Shorten this paragraph:

% o

RCS/ch01_summary,v  ->  ch01_summary

revision 1.3 (locked)

done

% v

"ch01_summary" 23 lines, 1243 characters

   ...

Typing a new h command stores a new filename.

If you always want to do the same commands on a file, you can store all the commands in one alias:

% alias d 'set f="\!*"; co -l "$f" && vi "$f" && ci "$f"'

% d ch01_summary

The && (two ampersands) (Section 35.14) means that the following command won't run unless the previous command returns a zero ("success") status. If you don't want that, use ; (semicolon) (Section 28.16) instead of &&.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's how to be more sure of your history before you use it. First, remember that the history substitutions !/ and !fra are replaced with the most recent command lines that started with / and fra, respectively.

If your memory is like mine (not very good), you might not be sure that !/ will bring back the command you want. You can test it by adding :p to the end. The shell will print the substitution but won't execute the command line. If you like what you got, type !! to execute it. For example:

# !/:p

/usr/sbin/sendmail -qv

# !!

/usr/sbin/sendmail -qv

Running id12345...

At the first prompt, the :p meant the command line was only printed. At the second prompt, I didn't use :p and the sendmail command was executed. The :p works with all history operators — for instance, !?sendmail?:p.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you want your command history to be remembered even when you log out, set the C shell's savehist shell variable (Section 35.9) to the number of lines of history you want saved. Other shells save history automatically; you don't need to set a variable. (If you want to change the number of lines saved by bash, set its HISTFILESIZE environment variable. In zsh , the variable is SAVEHIST. In ksh, the HISTSIZE variable sets the number of commands available to be recalled in your current shell as well the number saved for other shells.)

When you log out, the specified number of lines from the csh history list will be saved in a file called .history in your home directory. zsh, bash and ksh use the filename given in the HISTFILE environment variable. By default, bash calls the file .bash_history, and the original ksh uses .sh_history — but note that the new pdksh and zsh don't save history unless you set HISTFILE to a filename. For zsh, I chose $HOME/.zsh_history, but you can use anything you want.

On modern windowing systems, this isn't as trivial as it sounds. On an old-style terminal, people usually started only one main shell, so they could set the history-saving variable in their .login or .profile file and have it apply to their login shell.

However, under window systems like X or networked filesystems that share your home directory between several hosts, or on networked servers to which you might login via ssh, you may have multiple shells saving into the same history file. Linux systems with multiple virtual consoles (Section 23.12) logged on as the same user will have the same problem. The shells might be overwriting instead of appending, or appending instead of overwriting, or jumbling commands together when you want them separated. The sections below give some possible fixes.

Here's the basic way to give a separate history file to each

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most shells can save a history of the commands you type (Section 30.12). You can add your own commands to some shells' history lists without retyping them. Why would you do that?

You might have a set of commands that you want to be able to recall and reuse every time you log in. This can be more convenient than aliases because you don't have to think of names for the aliases. It's handier than a shell script if you need to do a series of commands that aren't always in the same order.
You might have several shells running (say, in several windows) and want to pass the history from one shell to another shell (Section 30.12).

Unfortunately, this isn't easy to do in all shells. For instance, the new pdksh saves its history in a file with NUL-separated lines. And the tcsh history file has a timestamp-comment before each saved line, like this:

#+0972337571

less 1928.sgm

#+0972337575

vi 1928.sgm

#+0972337702

ls -lt | head

Let's look at an example for two of the shells that make history editing easy. Use the csh command history -h, or the bash command history -w , to save the history from a shell to a file. Edit the file to take out commands you don't want:

% mail -s "My report" bigboss   $ mail -s "My report" bigboss

    ...                                           ...

% history -h > history.std      $ history -w history.std

% vi history.std                $ vi history.std

               ...Clean up history...                   

               ...Clean up history...

Read that file into another shell's history list with the csh command source -h or the bash command history -r :

% source -h history.std         $ history -r history.std

% !ma                           $ !ma

mail -s "My report" bigboss     mail -s "My report" bigboss

Of course, you can also use bash interactive command-line editing (Section 30.14) on the saved commands.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When Unix started, a lot of people used real teletypes — with a roll or box of paper instead of a glass screen. So there was no way to recall and edit previous commands. (You could see them on the paper, of course, but to reuse them you had to retype them.) The C shell added history substitution operators (Section 30.2) that were great on teletypes — and are still surprisingly useful on "glass teletypes" these days. All shells except the oldest Bourne shells still have history substitution, although it's limited in the Korn shells.

Modern shells also have interactive command-line editing. That's the ability to recall previous command lines and edit them using familiar vi or emacs commands. Arrow keys — along with Backspace and DELETE keys — generally work, too. So you don't need to know emacs or vi to edit command lines. But — especially if you're comfortable with Emacs-style commands (meta-this that, control-foo bar) — you'll find that most shells let you do much more than simply editing command lines. Shells can automatically correct spelling, complete partially-typed filenames (Section 28.6), and much more.

The basic idea of command-line editing is that the previous commands are treated like lines in a plain-text file, with the most recently typed commands at the "end" of the file. By using the editor's "up line" commands (like k in vi or C-p in Emacs), you can bring copies of earlier commands under your cursor, where you can edit them or simply re-execute them without changes. (It's important to understand that you're not editing the original commands; you're editing copies of them. You can recall a previous command as many times as you want to; its original version won't be changed as you edit the copy.) When you've got a command you want to run, you simply press ENTER; your cursor doesn't have to be at the end of the line. You can use CTRL-c (C-c in Emacs jargon) to cancel a command without running it and get a clean shell prompt.

It would be easy for us to fill a chapter with info on command-line editing. (In this book's fourth edition, maybe we should!) Because every shell has its own way to do this, though, we've decided to stick to the basics — with a few of the bells and whistles tossed in as examples. To really dig into this, check your shell's manpage or its Nutshell Handbook.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The existence of special characters (particularly !) can be a pain; you may often need to type commands that have exclamation points in them, and occasionally need commands with carets (^). These get the C shell confused unless you "quote" them properly. If you use these special characters often, you can choose different ones by setting the histchars variable. histchars is a two-character string; the first character replaces the exclamation point (the "history" character), and the second character replaces the caret (the "modification" character (Section 30.5)). For example:

% set histchars='@#'

% ls file*

file1   file2   file3

% @@                    

               Repeat previous command (was 

               !!

               )

ls file*

file1   file2   file3

% #file#data#           

               Edit previous command (was 

               ^file^data^

               )

ls data*

data4   data5

zsh's histchars is like the csh and tcsh version, but it has three characters. The third is the comment character — by default, #.

An obvious point: you can set histchars to any characters you like (provided they are different!), but it's a good idea to choose characters that you aren't likely to use often on command lines. Two good choices might be # (hash mark) and , (comma).

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you need to use ! (or your current history character) for a command (for example, if you still use uucp or send mail to someone who does, using the command-line mail ( Section 1.21) command), you can type a backslash (\) before each history character. You can also drop into the Bourne or Korn shell quickly — assuming that you aren't on a system that has replaced the real Bourne shell with a shell like bash that has history substitution built in. (If you're stuck, you can use the command set +H in bash; this disables history substitution.) Either of these are probably easier than changing histchars. For example:

% mail ora\!ishtar\!sally < file1    

               Quote the !s

% sh                                 

               Start the Bourne shell

$ mail ora!ishtar!sally < file1      

               ! not special here

$ exit                               

               Quit the Bourne shell

%                                    And back to the C shell

The original Bourne shell doesn't have any kind of history substitution, so ! doesn't mean anything special; it's just a regular character.

By the way, if you have a window system, you can probably copy and paste the command line (Section 28.10) instead of using shell history.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

How quickly can you move around the Unix filesystem? Can you locate any file or directory on your filesystem with both its absolute and relative pathnames? How can symbolic links help you and hurt you?

A lot of Unix users don't realize how much they'll be helped by completely understanding a few filesystem basics. Here are some of the most important concepts and tricks to know:

Using relative and absolute pathnames: Section 31.2.
What good is a current directory? Section 31.3.
Saving time and typing when changing directories with cdpath: Section 31.5.
Directory stacks keep a list of directories you're using and let you get to them quickly: Section 31.7, Section 31.8.
Quick cd aliases: Section 31.9.
Using variables and a tilde (~) to help you find directories and files: Section 31.11.
A mark alias to mark directory for cd'ing back: Section 31.12.
Automatic setup for entering and exiting a directory: Section 31.13.

— JP

Everything in the Unix filesystem — files, directories, devices, named pipes, and so on — has two pathnames: absolute and relative. If you know how to find those names, you'll know the best way to locate the file (or whatever) and use it. Even though pathnames are amazingly simple, they're one of the biggest problems beginners have. Studying this article carefully can save you a lot of time and frustration. See Figure 31-1 for an illustration of the Unix filesystem.

Figure 31-1: A Unix filesystem tree

Table 31-1 describes the two kinds of pathnames.

Table 31-1: Absolute and relative pathnames
Absolute pathnames	Relative pathnames
Start at the root directory.	Start at your current directory

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

How quickly can you move around the Unix filesystem? Can you locate any file or directory on your filesystem with both its absolute and relative pathnames? How can symbolic links help you and hurt you?

A lot of Unix users don't realize how much they'll be helped by completely understanding a few filesystem basics. Here are some of the most important concepts and tricks to know:

Using relative and absolute pathnames: Section 31.2.
What good is a current directory? Section 31.3.
Saving time and typing when changing directories with cdpath: Section 31.5.
Directory stacks keep a list of directories you're using and let you get to them quickly: Section 31.7, Section 31.8.
Quick cd aliases: Section 31.9.
Using variables and a tilde (~) to help you find directories and files: Section 31.11.
A mark alias to mark directory for cd'ing back: Section 31.12.
Automatic setup for entering and exiting a directory: Section 31.13.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Everything in the Unix filesystem — files, directories, devices, named pipes, and so on — has two pathnames: absolute and relative. If you know how to find those names, you'll know the best way to locate the file (or whatever) and use it. Even though pathnames are amazingly simple, they're one of the biggest problems beginners have. Studying this article carefully can save you a lot of time and frustration. See Figure 31-1 for an illustration of the Unix filesystem.

Figure 31-1: A Unix filesystem tree

Table 31-1 describes the two kinds of pathnames.

Table 31-1: Absolute and relative pathnames
Absolute pathnames	Relative pathnames
Start at the root directory.	Start at your current directory (Section 1.16).
Always start with a slash (`/`).	Never start with a slash.
The absolute pathname to some object (file, etc.) is always the same.	The relative pathname to an object depends on your current directory.

To make an absolute pathname:

Start at the root directory (/) and work down.
Put a slash (/) after every directory name — though if the path ends at a directory, the slash after the last name is optional.

For example, to get a listing of the directory highlighted in Figure 31-1, no matter what your current directory is, you'd use an absolute pathname like this:

% ls /home/jane/data

Sub    a    b     c

To make a relative pathname:

Start at your current directory.
As you move down the tree, away from root, add subdirectory names.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

People who think the cd command is all they need to know about current directories should read this article! Understanding how Unix uses the current directory can save you work.

Each Unix process has its own current directory. For instance, your shell has a current directory. So do vi, ls, sed, and every other Unix process. When your shell starts a process running, that child process starts with the same current directory as its parent. So how does ls know which directory to list? It uses the current directory it inherited from its parent process, the shell:

% ls

               ...Listing of ls's current directory appears,

         which is the same current directory as the shell.

Each process can change its current directory and that won't change the current directory of other processes that are already running. So:

Your shell script (which runs in a separate process) can cd to another directory without affecting the shell that started it. (So, the script doesn't need to cd back to the directory where it started before it exits.)
If you have more than one window or login session to the same computer, they probably run separate processes. So, they have independent current directories.
When you use a subshell (Section 43.7, Section 24.4) or a shell escape, you can cd anywhere you want. After you exit that shell, the parent shell's current directory won't have changed. For example, if you want to run a command in another directory without cding there first (and having to cd back), do it in a subshell:
```
% pwd

/foo/bar

% (cd 

                     baz; somecommand

                      > somefile)

% pwd

/foo/bar
```

When you really get down to it, what good is a current directory? Here it is: relative pathnames start at the current directory. Having a current directory means you can refer to a file by its relative pathname, like afile. Programs like ls access the current directory through its relative pathname . (dot) ( Section 1.16). Without a current directory and relative pathnames, you'd always have to use

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[ This article is about the standard Unix pwd command, an external (Section 1.9) command that isn't built into your shell. (The external pwd is usually stored at /bin/pwd.) Most shells have an internal version of pwd that "keeps track" of you as you change your current directory; it doesn't have to search the filesystem to find the current directory name. This article describes how the external version finds the pathname of its current directory. This isn't just academic stuff: seeing how pwd finds its current directory should help you understand how the filesystem is put together. — JP]

A command like pwd inherits the current directory of the process that started it (usually a shell). It could be started from anywhere. How does pwd find out where it is in the filesystem? See Figure 31-2 for a picture of the current directory /usr/joe and its parent directories. The current directory doesn't contain its own name, so that doesn't help pwd. But it has an entry named . (dot), which gives the i-number of the directory (Section 10.2).

Figure 31-2: Finding the current directory name

The current directory has i-number 234. Next, pwd asks Unix to open the parent directory file, the directory one level up, through the relative pathname (..). It's looking for the name that goes with i-number 234. Aha: the current directory is named joe, so the end of the pathname must be joe.

Next step: pwd looks at the . entry in the directory one level up to get its i-number, 14. As always, the name of the one-level-up directory is in its parent (.., i-number 12). To get its name, pwd opens the directory two levels up and looks for i-number 14, usr. Now pwd has the pathname usr/joe.

Same steps: look in the parent, i-number 12. What's its name? Hmmm. The i-number of its parent, 12, is the same as its own — and there's only one directory on the filesystem like this: the root directory (/). So pwd adds a slash to the start of the pathname and it's done: /usr/joe.

This explanation is really missing one or two parts: filesystems can be mounted on other filesystems, or they can be mounted across the network from other hosts. So at each step,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some people make a shell alias (Section 29.2) for directories they cd to often. Other people set shell variables (Section 35.9) to hold the pathnames of directories they don't want to retype. But both of those methods make you remember directory abbreviations — and make you put new aliases or shell variables in your shell startup files (Section 3.3) each time you want to add or change one. There's another way: the C shell's cdpath shell variable and the CDPATH variable in ksh, bash, and some versions of sh. (zsh understands both cdpath and CDPATH.) I'll use the term "cdpath" to talk about all shells.

When you type the command cd foo, the shell first tries to go to the exact pathname foo. If that doesn't work, and if foo is a relative pathname, the shell tries the same command from every directory listed in the cdpath. (If you use ksh or sh, see the note at the end of this article.)

Let's say that your home directory is /home/lisa and your current directory is somewhere else. Let's also say that your cdpath has the directories /home/lisa, /home/lisa/projects, and /books/troff. If your cd foo command doesn't work in your current directory, your shell will try

cd

/home/lisa/

foo,

cd

/home/lisa/projects/

foo, and cd /books/troff/ foo, in that order. If the shell finds one, it shows the pathname:

% cd foo

/home/lisa/foo

%

If there is more than one matching directory, the shell uses the first match; if this isn't what you wanted, you can change the order of the directories in the cdpath.

Some Bourne shells don't show the directory name. All shells print an error, though, if they can't find any foo directory.

So, set your cdpath to a list of the parent directories that contain directories you might want to cd to. Don't list the exact directories — list the parent directories (Section 1.16). This list goes in your .tcshrc, .cshrc, or .profile file. For example,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Normally a for loop (Section 35.21) iterates until it has processed all its word arguments. while and until loops (Section 35.15) iterate until the loop control command returns a certain status. But sometimes — for instance, if there's an error — you want a loop to immediately terminate or jump to the next iteration. That's where you use break and continue, respectively.

break terminates the loop and takes control to the line after done. continue skips the rest of the commands in the loop body and starts the next iteration. Here's an example of both. An outer loop is stepping through a list of directories. If we can't cd to one of them, we'll abort the loop with break. The inner loop steps through all entries in the directory. If one of the entries isn't a file or isn't readable, we skip it and try the next one.

'...' Section 28.14, || Section 35.14, * Section 1.13, test Section 35.26

for dir in `find $HOME/projdir -type d -print`

do

    cd "$dir" || break

    echo "Processing $dir"

    for file in *

    do

        test -f "$file" -a -r "$file" || continue

            ...process $dir/$file...

    done

done

With nested loops (like the file loop above), which loop is broken or continued? It's the loop being processed at that time. So, the continue here restarts the inner (file) loop. The break terminates the outer (directory) loop — which means the inner loop is also terminated. Note also that the -print argument to find is often redundant in the absence of another expression, depending on your version of find.

Here we've used break and continue within for loops, after the shell's || operator. But you can use them anywhere within the body of any loop — in an if statement within a while loop, for instance.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

How often do you need to move to some other directory temporarily, look at some file, and then move back to the directory where you started? If you're like most users, you do this all the time. Most shells have pushd and popd commands to make this a lot easier. (If you use the original ksh , Learning the Korn Shell, by Bill Rosenblatt and Arnold Robbins and also published by O'Reilly, shows you shell functions that do the same thing.)

These commands implement a "directory stack." The classical analogy for a stack is one of those spring-loaded plate stackers in a school (or corporate) cafeteria. The last plate put ("pushed") onto the stack is the first plate taken ("popped") from the stack. It's just the same with directories: each time you use pushd, the shell adds your current directory to the stack and moves you to the new directory. When you use popd, the shell takes the top directory off the stack and moves you to the directory underneath.

You may as well learn about pushd the way I did: by watching. Let's say that I'm in the directory ~/power, working on this book. I want to change to my Mail directory briefly, to look at some old correspondence. Let's see how. (Note that if you have a cdpath (Section 31.5) that includes your home directory, ~ or $HOME, you won't need to type the ~/ with arguments to pushd. In other words, pushd looks at your cdpath.)

los% pushd ~/Mail   

               current directory becomes ~/Mail

~/Mail ~/power

pushd prints the entire stack, giving me some confirmation about where I am and where I can go. When I'm done reading the old mail, I want to move back:

los% popd   

                               current directory becomes ~/power

~/power

We're back where we started; the Mail directory is no longer on the stack.

What if you want to move back and forth repeatedly? pushd, with no arguments, just switches the two top directories on the stack, like this:

los% pwd   

                                 current directory is ~/power

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The pushd command (Section 31.7) is nice for jumping around the filesystem, but some of the commands you might type a lot, like pushd +4, are sort of a pain to type. I saw these aliases (Section 29.2) in Daniel Gilly's setup file. They looked so handy that I decided to steal them for this book. There are C shell versions in the first column and Bourne-type in the second:

alias pd pushd                 alias pd=pushd

alias pd2 'pushd +2'           alias pd2='pushd +2'

alias pd3 'pushd +3'           alias pd3='pushd +3'

alias pd4 'pushd +4'           alias pd4='pushd +4'

      ...

So, for example, to swap the fourth directory on the stack, just type pd4.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you do a lot of work in some particular directories, it can be handy to make aliases (Section 29.2) that take you to each directory quickly. For example, this Korn shell alias lets you type pwr to change to the /books/troff/pwrtools directory:

alias pwr='cd /books/troff/pwrtools'

(If your shell doesn't have aliases, you can use a shell function (Section 29.11). A shell script (Section 1.8) won't work, though, because it runs in a subshell (Section 24.4).)

When you pick the alias names, it's a good idea not to conflict with command names that are already on the system. Section 35.27 shows how to pick a new name.

If you have a lot of these directory-changing aliases, you might want to put them in a separate file named something like .cd_aliases. Then add these lines to your shell setup file (Section 3.3), like this C shell example for your .cshrc :

source Section 35.29, ~ Section 31.11

alias setcds source ~/.cd_aliases

setcds

That reads your aliases into each shell. If you edit the .cd_aliases file, you can read the new file into your shell by typing setcds from any directory.

Finally, if you're in a group of people who all work on the same directories, you could make a central alias file that everyone reads from their shell setup files as they log in. Just adapt the example above.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's a handy shell function called c for people who cd all over the filesystem. (I first saw Marc Brumlik's posting of it on Usenet years ago, as a C shell alias. He and I have both made some changes to it since then.) This function is great for shells that don't have filename completion (Section 28.6). This function works a bit like filename completion, but it's faster because the "initials" match only directories and you don't have to press TAB or ESC after each part of the pathname. Instead, you just type the initials (first letter or more) of each directory in the pathname. Start at the root directory. Put a dot (.) after each part.

Here are three examples. The first one shows that there's no subdirectory of root whose name starts with q. The second one matches the directory /usr/include/hsfs and cds there:

$ c q.

c: no match for /q*/.

$ c u.i.h.

/usr/include/hsfs/.

$

In the next example, trying to change to /usr/include/pascal the abbreviations aren't unique the first time. The function shows me all the matches; the second time, I add another letter ("a") to make the name unique:

$ c u.i.p.

c: too many matches for u.i.p.:

/usr/include/pascal/. /usr/include/pixrect/. /usr/include/protocols/.

$ c u.i.pa.

/usr/include/pascal/.

$

Go to http://examples.oreilly.com/upt3 for more information on: c.csh, c.sh

The Bourne shell function is straightforward; it's shown below. The C shell alias needs some trickery, and there are two versions of it: one if you already have an alias for cd and another if you don't. (The C shell if used in the c alias won't work with a cd alias. Although the csh manual page admits it won't work, I'd call that another C shell bug.)

set Section 35.25, $# Section 35.20

function c( )

{

   dir="$1"



   # Delete dots.  Surround every letter with "/" and "*".

   # Add a final "/." to be sure this only matches a directory:

   dirpat="`echo $dir | sed 's/\([^.]*\)\./\/\1*/g'`/."



   # In case $dirpat is empty, set dummy "x" then shift it away:

   set x $dirpat; shift



   # Do the cd if we got one match, else print error:

   if [ "$1" = "$dirpat" ]; then

      # pattern didn't match (shell didn't expand it)

      echo "c: no match for $dirpat" 1>&2

   elif [ $# = 1 ]; then

      echo "$1"

      cd "$1"

   else

      echo "c: too many matches for $dir:" 1>&2

      ls -d "$@"

   fi



   unset dir dirpat

}

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most shells have a shortcut for the pathname to your home directory: a tilde (~), often called "twiddle" by Unix-heads. You can use ~ in a pathname to the home directory from wherever you are. For example, from any directory, you can list your home directory or edit your .cshrc file in it by typing:

% ls ~

   ...

% vi ~/.cshrc

If you're using a very old Bourne shell, one that does not support the tilde convention, try the $HOME or $LOGDIR variables instead.

You could change your current directory to your home directory by typing cd ~ or cd $HOME, but all shells have a shorter shortcut: typing plain cd with no argument also takes you home.

If your shell understands the tilde, it should also have an abbreviation for other users' home directories: a tilde with the username on the end. For example, the home directory for mandi, which might really be /remote/users/m/a/mandi, could be abbreviated ~mandi. On your account, if Mandi told you to copy the file named menu.c from her src directory, you could type:

% cp ~mandi/src/menu.c .

Don't confuse this with filenames like report~. Some programs, like the GNU Emacs (Section 19.4) editor and vi, may create temporary filenames that end with a ~ (tilde).

Your version of the Bourne shell might also emulate the special "directory" /u — if your system administrator hasn't already set up /u, that is. It's a directory full of symbolic links (Section 10.4) to users' home directories. For instance, /u/jane could be a link to /home/users/jane. Many systems are now using /home for home directories, in favor of the old /usr/users or /u conventions. Darwin uses /Users/username (note the uppercase U!), but the tilde works the same there, too.

If all else fails, here's a trick that's probably too ugly to type a lot, but it's useful in Bourne shell scripts, where you don't want to "hardcode" users' home directory pathnames. This command calls the C shell to put

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The following alias stores the current directory name in a variable:

alias mark 'set \!:1=$cwd'

so as to use a feature of the C shell:

% mark 

               here

   ...

% cd 

               here

One need not even type $ here. If a directory does not exist, csh tries searching its cdpath (Section 31.5), then tries to evaluate the name as a variable.

(I generally use pushd and popd (Section 31.7) to store directory names; mark is more useful with commands that need to look in two different paths, and in that case $ here is necessary anyway. Ah well.)

[In bash and zsh, you can do this by setting cdable_vars . In your shell setup file (Section 3.3), use cdable_vars=1 for bash or setopt cdable_vars or setopt -T for zsh. — JP]

— CT

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you work in a lot of different directories, here's a way to make the shell do automatic setup when you enter a directory or do cleanup as you leave. We've broken it onto two lines for printing; enter it as one line. On bash, make a shell function instead.

Go to http://examples.oreilly.com/upt3 for more information on: csh_init, sh_init

alias cd 'if (-o .exit.csh) source .exit.csh; chdir \!*;

    if (-o .enter.csh) source .enter.csh'



function cd( ) {

   test -r .exit.sh && . .exit.sh

   builtin cd "$1"                  # builtin is a bash command

   test -r .enter.sh && . .enter.sh

}

Then create .enter.csh and/or .exit.csh files in the directories where you want a custom setup. Bourne-type shell users, make .enter.sh and/or .exit.sh files instead. When you cd to a new directory, an .exit file is sourced (Section 35.29) into your current shell before you leave the old directory. As you enter the new directory, a .enter file will be read if it exists. If you use pushd and popd (Section 31.7), you'll probably want to make the same kind of aliases or functions for them.

The C shell alias tests to be sure you own the files; this helps to stop other users from leaving surprises for you! But if lots of users will be sharing the directory, they may all want to share the same files — in that case, replace the -o tests with -r (true if the file is readable).

Here's a sample .enter.csh file:

Go to http://examples.oreilly.com/upt3 for more information on: .enter.csh, .enter.sh

# Save previous umask; reset in .exit.csh:

set prevumask=`umask`



# Let everyone in the group edit my files here:

umask 002

echo ".enter.csh: setting umask to 002"

# Prompt (with blank line before) to keep me awake:

set prompt="\

$cwd - PROJECT DEVELOPMENT DIRECTORY.  EDIT CAREFULLY...\

% "

Here's the .exit.csh to go with it:

Go to http://examples.oreilly.com/upt3 for more information on: .exit.csh, .exit.sh

setprompt

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When my young daughter is struggling to understand the meaning of an idiomatic expression, such as, "Someone let the cat out of the bag," before I tell her what it means, I have to tell her that it's an expression, that she's not to interpret it literally. (As a consequence, she also uses "That's just an expression" to qualify her own remarks, especially when she is unsure about what she has just said.)

An expression, even in computer terminology, is not something to be interpreted literally. It is something that needs to be evaluated.

Many Unix programs use a special "regular expression syntax" for specifying what you could think of as "wildcard searches" through files. Regular expressions describe patterns, or sequences of characters, without necessarily specifying the characters literally. You'll also hear this process referred to as " pattern matching."

In this chapter, we depart a bit from the usual "tips and tricks" style of the book to provide an extended tutorial about regular expressions that starts in Section 32.4. We did this because regular expressions are so important to many of the tips and tricks elsewhere in the book, and we wanted to make sure that they are covered thoroughly.

This tutorial article is accompanied by a few snippets of advice (Section 32.16 and Section 32.18) and a few tools that help you see what your expressions are matching (Section 32.17). There's also a quick reference (Section 32.21) for those of you who just need a refresher.

For tips, tricks, and tools that rely on an understanding of regular expression syntax, you have only to look at:

Chapter 13
Chapter 17
Chapter 20
Chapter 34
Chapter 41

O'Reilly's Mastering Regular Expressions, by Jeffrey Friedl, is a gold mine of examples and specifics.

—DD and TOR

Before we even start talking about regular expressions, a word of caution for beginners: regular expressions can be confusing because they look a lot like the file-matching patterns ("wildcards") the shell uses. Both the shell and programs that use regular expressions have special meanings for the asterisk (

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When my young daughter is struggling to understand the meaning of an idiomatic expression, such as, "Someone let the cat out of the bag," before I tell her what it means, I have to tell her that it's an expression, that she's not to interpret it literally. (As a consequence, she also uses "That's just an expression" to qualify her own remarks, especially when she is unsure about what she has just said.)

An expression, even in computer terminology, is not something to be interpreted literally. It is something that needs to be evaluated.

Many Unix programs use a special "regular expression syntax" for specifying what you could think of as "wildcard searches" through files. Regular expressions describe patterns, or sequences of characters, without necessarily specifying the characters literally. You'll also hear this process referred to as " pattern matching."

In this chapter, we depart a bit from the usual "tips and tricks" style of the book to provide an extended tutorial about regular expressions that starts in Section 32.4. We did this because regular expressions are so important to many of the tips and tricks elsewhere in the book, and we wanted to make sure that they are covered thoroughly.

This tutorial article is accompanied by a few snippets of advice (Section 32.16 and Section 32.18) and a few tools that help you see what your expressions are matching (Section 32.17). There's also a quick reference (Section 32.21) for those of you who just need a refresher.

For tips, tricks, and tools that rely on an understanding of regular expression syntax, you have only to look at:

Chapter 13
Chapter 17
Chapter 20
Chapter 34
Chapter 41

O'Reilly's Mastering Regular Expressions, by Jeffrey Friedl, is a gold mine of examples and specifics.

—DD and TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Before we even start talking about regular expressions, a word of caution for beginners: regular expressions can be confusing because they look a lot like the file-matching patterns ("wildcards") the shell uses. Both the shell and programs that use regular expressions have special meanings for the asterisk (*), question mark (?), parentheses (( )), square brackets (

[

]

), and vertical bar (|, the "pipe").

Some of these characters even act the same way — almost.

Just remember, the shells, find, and some others generally use filename-matching patterns and not regular expressions.

You also have to remember that shell wildcards are expanded before the shell passes the arguments to the program. To prevent this expansion, the special characters in a regular expression must be quoted (Section 27.12) when passed as an argument from the shell.

The command:

$ grep [A-Z]*.c chap[12]

could, for example, be interpreted by the shell as:

grep Array.c Bug.c Comp.c chap1 chap2

and so grep would then try to find the pattern "Array.c" in files Bug.c, Comp.c, chap1, and chap2.

The simplest solution in most cases is to surround the regular expression with single quotes ('). Another is to use the echo command to echo your command line to see how the shell will interpret the special characters.

—BB and DG, TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You are probably familiar with the kinds of expressions that a calculator interprets. Look at the following arithmetic expression:

2 + 4

"Two plus four" consists of several constants or literal values and an operator. A calculator program must recognize, for instance, that 2 is a numeric constant and that the plus sign represents an operator, not to be interpreted as the + character.

An expression tells the computer how to produce a result. Although it is the sum of "two plus four" that we really want, we don't simply tell the computer to return a six. We instruct the computer to evaluate the expression and return a value.

An expression can be more complicated than 2+4; in fact, it might consist of multiple simple expressions, such as the following:

2 + 3 * 4

A calculator normally evaluates an expression from left to right. However, certain operators have precedence over others: that is, they will be performed first. Thus, the above expression evaluates to 14 and not 20 because multiplication takes precedence over addition. Precedence can be overridden by placing the simple expression in parentheses. Thus, (2+3)*4 or "the sum of two plus three times four" evaluates to 20. The parentheses are symbols that instruct the calculator to change the order in which the expression is evaluated.

A regular expression, by contrast, is descriptive of a pattern or sequence of characters. Concatenation is the basic operation implied in every regular expression. That is, a pattern matches adjacent characters. Look at the following example of a regular expression:

ABE

Each literal character is a regular expression that matches only that single character. This expression describes "an A followed by a B followed by an E" or simply the string ABE. The term "string" means each character concatenated to the one preceding it. That a regular expression describes a sequence of characters can't be emphasized enough. (Novice users are inclined to think in higher-level units such as words, and not individual characters.) Regular expressions are case-sensitive;

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are three important parts to a regular expression:

Anchors: Specify the position of the pattern in relation to a line of text.
Character sets: Match one or more characters in a single position.
Modifiers: Specify how many times the previous character set is repeated.

The following regular expression demonstrates all three parts:

^#*

The caret (^) is an anchor that indicates the beginning of the line. The hash mark is a simple character set that matches the single character #. The asterisk (*) is a modifier. In a regular expression, it specifies that the previous character set can appear any number of times, including zero. As you will see shortly, this is a useless regular expression (except for demonstrating the syntax!).

There are two main types of regular expressions: simple (also known as basic) regular expressions and extended regular expressions. (As we'll see in the next dozen articles, the boundaries between the two types have become blurred as regular expressions have evolved.) A few utilities like awk and egrep use the extended regular expression. Most use the simple regular expression. From now on, if I talk about a "regular expression" (without specifying simple or extended), I am describing a feature common to both types. For the most part, though, when using modern tools, you'll find that extended regular expressions are the rule rather than the exception; it all depends on who wrote the version of the tool you're using and when, and whether it made sense to worry about supporting extended regular expressions.

[The situation is complicated by the fact that simple regular expressions have evolved over time, so there are versions of "simple regular expressions" that support extensions missing from extended regular expressions! Bruce explains the incompatibility at the end of Section 32.15. —

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most Unix text facilities are line-oriented. Searching for patterns that span several lines is not easy to do. [But it is possible (Section 13.9, Section 11.10). — JP] You see, the end-of-line character is not included in the block of text that is searched. It is a separator, and regular expressions examine the text between the separators. If you want to search for a pattern that is at one end or the other, you use anchors. The caret (^) is the starting anchor, and the dollar sign ($) is the end anchor. The regular expression ^A will match all lines that start with an uppercase A. The expression A$ will match all lines that end with uppercase A. If the anchor characters are not used at the proper end of the pattern, they no longer act as anchors. That is, the ^ is an anchor only if it is the first character in a regular expression. The $ is an anchor only if it is the last character. The expression $1 does not have an anchor. Neither does 1^. If you need to match a ^ at the beginning of the line or a $ at the end of a line, you must escape the special character by typing a backslash (\) before it. Table 32-1 has a summary.

Table 32-1: Regular expression anchor character examples
Pattern	Matches
`^A`	An `A` at the beginning of a line
`A$`	An `A` at the end of a line
`A`	An `A` anywhere on a line

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The simplest character set is a single character. The regular expression the contains three character sets: t, h, and e. It will match any line that contains the string the, including the word other. To prevent this, put spaces (·) before and after the pattern: ·the·.

You can combine the string with an anchor. The pattern ^From:· will match the lines of a mail message (Section 1.21) that identify the sender. Use this pattern with grep to print every address in your incoming mailbox. [If your system doesn't define the environment variable MAIL, try /var/spool/mail/$USER or possibly /usr/spool/mail/$USER. — SJC]

$USER Section 35.5

% grep '^From: ' $MAIL

Some characters have a special meaning in regular expressions. If you want to search for such a character as itself, escape it with a backslash (\).

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The dot (.) is one of those special metacharacters. By itself it will match any character except the end-of-line character. The pattern that will match a line with any single character is ^.$.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you want to match specific characters, you can use square brackets, [ ], to identify the exact characters you are searching for. The pattern that will match any line of text that contains exactly one digit is ^[0123456789]$. This is longer than it has to be. You can use the hyphen between two characters to specify a range: ^[0-9]$. You can intermix explicit characters with character ranges. This pattern will match a single character that is a letter, digit, or underscore: [A-Za-z0-9_]. Character sets can be combined by placing them next to one another. If you wanted to search for a word that:

started with an uppercase T,
was the first word on a line,
had a lowercase letter as its second letter,
was three letters long (followed by a space character (·)), and
had a lowercase vowel as its third letter,

the regular expression would be:

^T[a-z][aeiou]·

To be specific: a range is a contiguous series of characters, from low to high, in the ASCII character set. For example, [z-a] is not a range because it's backwards. The range [A-z] matches both uppercase and lowercase letters, but it also matches the six characters that fall between uppercase and lowercase letters in the ASCII chart: [, \, ], ^, _, and '.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can easily search for all characters except those in square brackets by putting a caret (^) as the first character after the left square bracket ([). To match all characters except lowercase vowels, use [^aeiou].

Like the anchors in places that can't be considered an anchor, the right square bracket (]) and dash (-) do not have a special meaning if they directly follow a [. Table 32-2 has some examples.

Table 32-2: Regular expression character set examples
Regular expression	Matches
`[0-9]`	Any digit
`[^0-9]`	Any character other than a digit
`[-0-9]`	Any digit or a `-`
`[0-9-]`	Any digit or a `-`
`[^-0-9]`	Any character except a digit or a `-`
`[ ]0-9]`	Any digit or a `]`
`[0-9]]`	Any digit followed by a `]`

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The third part of a regular expression is the modifier. It is used to specify how many times you expect to see the previous character set. The special character * (asterisk) matches zero or more copies. That is, the regular expression 0* matches zero or more zeros, while the expression [0-9]* matches zero or more digits.

This explains why the pattern ^#* is useless (Section 32.4), as it matches any number of #s at the beginning of the line, including zero. Therefore, this will match every line, because every line starts with zero or more #s.

At first glance, it might seem that starting the count at zero is stupid. Not so. Looking for an unknown number of characters is very important. Suppose you wanted to look for a digit at the beginning of a line, and there may or may not be spaces before the digit. Just use ^·* to match zero or more spaces at the beginning of the line. If you need to match one or more, just repeat the character set. That is, [0-9]* matches zero or more digits and [0-9][0-9]* matches one or more digits.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You cannot specify a maximum number of sets with the * modifier. However, some programs (Section 32.20) recognize a special pattern you can use to specify the minimum and maximum number of repeats. This is done by putting those two numbers between \{ and \}.

Having convinced you that \{ isn't a plot to confuse you, an example is in order. The regular expression to match four, five, six, seven, or eight lowercase letters is:

[a-z]\{4,8\}

Any numbers between 0 and 255 can be used. The second number may be omitted, which removes the upper limit. If the comma and the second number are omitted, the pattern must be duplicated the exact number of times specified by the first number.

The backslashes deserve a special discussion. Normally a backslash turns off the special meaning for a character. For example, a literal period is matched by \. and a literal asterisk is matched by \*. However, if a backslash is placed before a <, >, {, }, (, or ) or before a digit, the backslash turns on a special meaning. This was done because these special functions were added late in the life of regular expressions. Changing the meaning of {, }, (, ), <, and > would have broken old expressions. (This is a horrible crime punishable by a year of hard labor writing COBOL programs.) Instead, adding a backslash added functionality without breaking old programs. Rather than complain about the change, view it as evolution.

You must remember that modifiers like * and \{1,5\} act as modifiers only if they follow a character set. If they were at the beginning of a pattern, they would not be modifiers. Table 32-3 is a list of examples and the exceptions.

Table 32-3: Regular expression pattern repetition examples
Regular expression	Matches
`*`	Any line with a `*`

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Searching for a word isn't quite as simple as it at first appears. The string the will match the word other. You can put spaces before and after the letters and use this regular expression: ·the·. However, this does not match words at the beginning or the end of the line. And it does not match the case where there is a punctuation mark after the word.

There is an easy solution — at least in many versions of ed, ex, vi, and grep. The characters \< and \> are similar to the ^ and $ anchors, as they don't occupy a position of a character. They anchor the expression between to match only if it is on a word boundary. The pattern to search for the words the and The would be: \<[tT]he\>.

Let's define a "word boundary." The character before the t or T must be either a newline character or anything except a letter, digit, or underscore ( _ ). The character after the e must also be a character other than a digit, letter, or underscore, or it could be the end-of-line character.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Another pattern that requires a special mechanism is searching for repeated words. The expression [a-z][a-z] will match any two lowercase letters. If you wanted to search for lines that had two adjoining identical letters, the above pattern wouldn't help. You need a way to remember what you found and see if the same pattern occurs again. In some programs, you can mark part of a pattern using $ and $. You can recall the remembered pattern with \ followed by a single digit. Therefore, to search for two identical letters, use $[a-z]$\1. You can have nine different remembered patterns. Each occurrence of $ starts a new pattern. The regular expression to match a five-letter palindrome (e.g., "radar") is: \([a-z]$$[a-z]$[a-z]\2\1. [Some versions of some programs can't handle  in the same regular expression as \1, etc. In all versions of sed, you're safe if you use  on the pattern side of an s command — and \1, etc., on the replacement side (Section 34.11). — JP]

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Before I discuss the extensions that extended expressions (Section 32.15) offer, I want to mention two potential problem areas.

The \< and \> characters were introduced in the vi editor. The other programs didn't have this ability at that time. Also, the \{ min,max \} modifier is new, and earlier utilities didn't have this ability. This makes it difficult for the novice user of regular expressions, because it seems as if each utility has a different convention. Sun has retrofitted the newest regular expression library to all of their programs, so they all have the same ability. If you try to use these newer features on other vendors' machines, you might find they don't work the same way.

The other potential point of confusion is the extent of the pattern matches (Section 32.17). Regular expressions match the longest possible pattern. That is, the regular expression A.*B matches AAB as well as AAAABBBBABCCCCBBBAAAB. This doesn't cause many problems using grep, because an oversight in a regular expression will just match more lines than desired. If you use sed, and your patterns get carried away, you may end up deleting or changing more than you want to. Perl answers this problem by defining a variety of "greedy" and "non-greedy" regular expressions, which allow you to specify which behavior you want. See the perlre(1) manual page for details.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

At least two programs use extended regular expressions: egrep and awk. [perl uses expressions that are even more extended. — JP] With these extensions, special characters preceded by a backslash no longer have special meaning: \{, \}, \<, \>, $, $, as well as \ digit. There is a very good reason for this, which I will delay explaining to build up suspense.

The question mark (?) matches zero or one instance of the character set before it, and the plus sign (+) matches one or more copies of the character set. You can't use \{ and \} in extended regular expressions, but if you could, you might consider ? to be the same as \{0,1\} and + to be the same as \{1,\}.

By now, you are wondering why the extended regular expressions are even worth using. Except for two abbreviations, there seem to be no advantages and a lot of disadvantages. Therefore, examples would be useful.

The three important characters in the expanded regular expressions are (, |, and ). Parentheses are used to group expressions; the vertical bar acts an an OR operator. Together, they let you match a choice of patterns. As an example, you can use egrep to print all From: and Subject: lines from your incoming mail [which may also be in /var/spool/mail/$USER. — JP]:

% egrep '^(From|Subject): ' /usr/spool/mail/$USER

All lines starting with From: or Subject: will be printed. There is no easy way to do this with simple regular expressions. You could try something like ^[FS][ru][ob][mj]e*c*t*: and hope you don't have any lines that start with Sromeet:. Extended expressions don't have the \< and \> characters. You can compensate by using the alternation mechanism. Matching the word "the" in the beginning, middle, or end of a sentence or at the end of a line can be done with the extended regular expression

(^|

)the([^a-z]|$)

. There are two choices before the word: a space or the beginning of a line. Following the word, there must be something besides a lowercase letter or else the end of the line. One extra bonus with extended regular expressions is the ability to use the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Writing regular expressions involves more than learning the mechanics. You not only have to learn how to describe patterns, but you also have to recognize the context in which they appear. You have to be able to think through the level of detail that is necessary in a regular expression, based on the context in which the pattern will be applied.

The same thing that makes writing regular expressions difficult is what makes writing them interesting: the variety of occurrences or contexts in which a pattern appears. This complexity is inherent in language itself, just as you can't always understand an expression (Section 32.1) by looking up each word in the dictionary.

The process of writing a regular expression involves three steps:

Knowing what you want to match and how it might appear in the text.
Writing a pattern to describe what you want to match.
Testing the pattern to see what it matches.

This process is virtually the same kind of process that a programmer follows to develop a program. Step 1 might be considered the specification, which should reflect an understanding of the problem to be solved as well as how to solve it. Step 2 is analogous to the actual coding of the program, and step 3 involves running the program and testing it against the specification. Steps 2 and 3 form a loop that is repeated until the program works satisfactorily.

Testing your description of what you want to match ensures that the description works as expected. It usually uncovers a few surprises. Carefully examining the results of a test, comparing the output against the input, will greatly improve your understanding of regular expressions. You might consider evaluating the results of a pattern-matching operation as follows:

Hits: The lines that I wanted to match.
Misses: The lines that I didn't want to match.
Misses that should be hits: The lines that I didn't match but wanted to match.
Hits that should be misses: The lines that I matched but didn't want to match.

Trying to perfect your description of a pattern is something that you work at from opposite ends: you try to eliminate the "hits that should be misses" by limiting the possible matches, and you try to capture the "misses that should be hits" by expanding the possible matches.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the toughest things to learn about regular expressions is just what they do match. The problem is that a regular expression tends to find the longest possible match — which can be more than you want.

Go to http://examples.oreilly.com/upt3 for more information on: showmatch

Here's a simple script called showmatch that is useful for testing regular expressions, when writing sed scripts, etc. Given a regular expression and a filename, it finds lines in the file matching that expression, just like grep, but it uses a row of carets (^^^^) to highlight the portion of the line that was actually matched. Depending on your system, you may need to call nawk instead of awk; most modern systems have an awk that supports the syntax introduced by nawk, however.

#! /bin/sh

# showmatch - mark string that matches pattern

pattern=$1; shift

awk 'match($0,pattern) > 0 {

    s = substr($0,1,RSTART-1)

    m = substr($0,1,RLENGTH)

    gsub (/[^\b- ]/, " ", s)

    gsub (/./,       "^", m)

    printf "%s\n%s%s\n", $0, s, m

}' pattern="$pattern" $*

For example:

% showmatch 'CD-...' mbox

and CD-ROM publishing. We have recognized

    ^^^^^^

that documentation will be shipped on CD-ROM; however,

                                      ^^^^^^

Go to http://examples.oreilly.com/upt3 for more information on: xgrep

xgrep is a related script that simply retrieves only the matched text. This allows you to extract patterned data from a file. For example, you could extract only the numbers from a table containing both text and numbers. It's also great for counting the number of occurrences of some pattern in your file, as shown below. Just be sure that your expression matches only what you want. If you aren't sure, leave off the wc command and glance at the output. For example, the regular expression [0-9]* will match numbers like 3.2 twice: once for the 3 and again for the 2! You want to include a dot (.) and/or comma (,), depending on how your numbers are written. For example:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A regular expression tries to match the longest string possible, which can cause unexpected problems. For instance, look at the following regular expression, which matches any number of characters inside quotation marks:

".*"

Let's imagine an HTML table with lots of entries, each of which has two quoted strings, as shown below:

<td><a href="#arts"><img src="d_arrow.gif" border=0></a>

All the text in each line of the table is the same, except the text inside the quotes. To match the line through the first quoted string, a novice might describe the pattern with the following regular expression:

<td><a href=".*">

However, the pattern ends up matching almost all of the entry because the second quotation mark in the pattern matches the last quotation mark on the line! If you know how many quoted strings there are, you can specify each of them:

<td><a href=".*"><img src=".*" border=0></a>

Although this works as you'd expect, some line in the file might not have the same number of quoted strings, causing misses that should be hits — you simply want the first argument. Here's a different regular expression that matches the shortest possible extent between two quotation marks:

"[^"]*"

It matches "a quote, followed by any number of characters that do not match a quote, followed by a quote." Note, however, that it will be fooled by escaped quotes, in strings such as the following:

$strExample = "This sentence contains an escaped \" character.";

The use of what we might call "negated character classes" like this is one of the things that distinguishes the journeyman regular expression user from the novice.

—DD and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Once you know regular expression syntax, you can match almost anything. But sometimes, it's a pain to think through how to get what you want. Table 32-4 lists some useful regular expressions that match various kinds of data you might have to deal with in the Unix environment. Some of these examples work in any program that uses regular expressions; others only work with a specific program such as egrep. (Section 32.20 lists the metacharacters that each program accepts.) The · means to use a space as part of the regular expression. Bear in mind that you may also be able to use \< and \> to match on word boundaries.

Note that these regular expressions are only examples. They aren't meant to match (for instance) every occurrence of a city and state in any arbitrary text. But if you can picture what the expression does and why, that should help you write an expression that fits your text.

Table 32-4: Some useful regular expressions
Item	Example	Regular expression
U.S. state abbreviation	(NM)	·[A-Z][A-Z]·
U.S. city, state	(Portland, OR)	^.*,·[A-Z][A-Z]
Month day, year	(JAN 05, 1993); (January 5, 1993)	[A-Z][A-Za-z]\{2,8\}·[0-9]\{1,2\},·[0-9]\{4\}
U.S. Social Security number	(123-45-6789)	[0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\}=

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some regular expression metacharacters are valid for one program but not for another. Those that are available to a particular Unix program are marked by a check (✓) in Table 32-5. Quick reference descriptions of each of the characters can be found in Section 32.21.

[Unfortunately, even this table doesn't give the whole story. For example, Sun has taken some of the extensions originally developed for ed, ex, and vi (such as the \< \> and \{ min, max \} modifiers) and added them to other programs that use regular expressions. So don't be bashful — try things out, but don't be surprised if every possible regular expression feature isn't supported by every program. In addition, there are many programs that recognize regular expressions, such as perl, emacs, more, dbx, expr, lex, pg, and less, that aren't covered in Daniel's table. — TOR]

Table 32-5: Valid metacharacters for different programs
Symbol	ed	ex	vi	sed	awk	grep	egrep	Action
.	✓	✓	✓	✓

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 32.4 gives an introduction to regular expressions. This article is intended for those of you who need just a quick listing of regular expression syntax as a refresher from time to time. It also includes some simple examples. The characters in Table 32-7 have special meaning only in search patterns.

Table 32-7: Special characters in search patterns
Pattern	What does it match?
.	Match any single character except newline.
`*`	Match any number (including none) of the single characters that immediately precede it. The preceding character can also be a regular expression. For example, since . (dot) means any character, `.*` means "match any number of any character."
`^`	Match the following regular expression at the beginning of the line.
`$`	Match the preceding regular expression at the end of the line.
`[ ]`	Match any one of the enclosed characters.
	A hyphen (`-`) indicates a range of consecutive characters. A caret (`^`) as the first character in the brackets reverses the sense: it matches any one character not in the list. A hyphen or a right square bracket (`]`) as the first character is treated as a member of the list. All other metacharacters are treated as members of the list.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Wildcards (Section 1.13) are the shell's way of abbreviating filenames. Just as in poker, where a wildcard is a special card that can match any card in the deck, filename wildcards are capable of matching letters or groups of letters in the alphabet. Rather than typing a long filename or a long chain of filenames, a wildcard lets you provide parts of names and then use some "wildcard characters" for the rest. For example, if you want to delete all files whose names end in .o, you can give the following command:

% rm *.o

You don't have to list every filename.

I'm sure you already know that wildcards are useful in many situations. If not, they are summarized in Section 33.2. Here are a few of my favorite wildcard applications:

If you remember part of a filename, but not the whole name, you can use wildcards to help you find it. If I have a file on genetics saved in a directory with several hundred other files, a command like:
```
% ls *gene*

                  
```
will often find what I want. It's quicker and easier than find (Section 9.1).
Wildcards are a natural when you want to work with groups of files. If I have a general purpose directory that's full of filenames ending in .c and .h, I can make new subdirectories and use wildcards to move the files easily:
```
% mkdir c h

% mv *.c c

                  
```
Wildcards often help you to work with files with inconvenient characters in their names. Let's say you have a file named abc x e, where x is some unknown control character. You can delete or rename that file by using the wildcarded name abc?e. (When you do this, be careful that your wildcard doesn't match more than you intend, perhaps by running an ls using the pattern first.)
Wildcards can appear in any component of a pathname. This can often be used to your advantage. For example, let's say that you have a directory named /work, split into subdirectories for a dozen different projects. For each project, you have a schedule, in a file called (obviously enough)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Wildcards (Section 1.13) are the shell's way of abbreviating filenames. Just as in poker, where a wildcard is a special card that can match any card in the deck, filename wildcards are capable of matching letters or groups of letters in the alphabet. Rather than typing a long filename or a long chain of filenames, a wildcard lets you provide parts of names and then use some "wildcard characters" for the rest. For example, if you want to delete all files whose names end in .o, you can give the following command:

% rm *.o

You don't have to list every filename.

I'm sure you already know that wildcards are useful in many situations. If not, they are summarized in Section 33.2. Here are a few of my favorite wildcard applications:

If you remember part of a filename, but not the whole name, you can use wildcards to help you find it. If I have a file on genetics saved in a directory with several hundred other files, a command like:
```
% ls *gene*

                  
```
will often find what I want. It's quicker and easier than find (Section 9.1).
Wildcards are a natural when you want to work with groups of files. If I have a general purpose directory that's full of filenames ending in .c and .h, I can make new subdirectories and use wildcards to move the files easily:
```
% mkdir c h

% mv *.c c

                  
```
Wildcards often help you to work with files with inconvenient characters in their names. Let's say you have a file named abc x e, where x is some unknown control character. You can delete or rename that file by using the wildcarded name abc?e. (When you do this, be careful that your wildcard doesn't match more than you intend, perhaps by running an ls using the pattern first.)
Wildcards can appear in any component of a pathname. This can often be used to your advantage. For example, let's say that you have a directory named /work, split into subdirectories for a dozen different projects. For each project, you have a schedule, in a file called (obviously enough)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This article summarizes the wildcards that are used for filename expansion (see Table 33-1). The shells use the same basic wildcards, though most shells have some extensions. Unless otherwise noted, assume that wildcards are valid for all shells.

Table 33-1: Filename wildcards
Wildcard	Shells	Description
`*`	All	Match zero or more characters. For example, `a*` matches the files a, ab, abc, abc.d, and so on. (zsh users: also see `x#` and `x##`, below.)
`?`	All	Match exactly one character. For example, `a?` matches aa, ab, ac, etc.
`[12..a..z]`	All	Match any character listed in the brackets. For example, `a[ab]` matches aa or ab.
`[a-z]`	All	Match all characters between a and z, in a case-sensitive manner, based on the characters' value in the ASCII character set. For example, `a[0-9]` matches a0, a1, and so on, up to a9.
`[!ab..z]`

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Wildcards (Section 1.13) are actually defined by the Unix shells, rather than the Unix filesystem. In theory, a new shell could define new wildcards, and consequently, we should discuss wildcarding when we discuss the shell. In practice, all Unix shells (including ksh, bash, and other variants (Section 1.6)) honor the same wildcard conventions, and we don't expect to see anyone change the rules. (But most new shells also have extended wildcards (Section 33.2). And different shells do different things when a wildcard doesn't match (Section 33.4).)

You may see different wildcarding if you have a special-purpose shell that emulates another operating system (for example, a shell that looks like the COMMAND.COM in MS-DOS) — in this case, your shell will obey the other operating system's wildcard rules. But even in this case, operating system designers stick to a reasonably similar set of wildcard rules.

The fact that the shell defines wildcards, rather than the filesystem itself or the program you're running, has some important implications for a few commands. Most of the time, a program never sees wildcards. For example, the result of typing:

% lpr *

is exactly the same as typing:

% lpr 

               file1 file2 file3 file4 file5

In this case everything works as expected. But there are other situations in which wildcards don't work at all. Assume you want to read some files from a tape, which requires the command tar x (Section 38.6), so you type the command tar x *.txt. Will you be happy or disappointed?

You'll be disappointed — unless older versions of the files you want are already in your current directory (Section 1.16). The shell expands the wildcard *.txt, according to what's in the current directory, before it hands the completed command line over to tar for execution. All tar gets is a list of files. But you're probably not interested in the current directory; you probably want the wildcard * to be expanded on the tape, retrieving any *.txt files that the tape has.

There's a way to pass wildcards to programs, without having them interpreted by the shell. Simply put

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I ran into a strange situation the other day. I was compiling a program that was core dumping. At some point, I decided to delete the object files and the core file, and start over, so I gave the command:

% rm *.o core

It works as expected most of the time, except when no object files exist. (I don't remember why I did this, but it was probably by using !! (Section 30.8) when I knew there weren't any .o's around.) In this case, you get No match, and the core file is not deleted.

It turns out, for C shell users, that if none of the wildcards can be expanded, you get a No match error. It doesn't matter that there's a perfectly good match for other name(s). That's because, when csh can't match a wildcard, it aborts and prints an error — it won't run the command. If you create one .o file or remove the *.o from the command line, core will disappear happily.

On the other hand, if the Bourne shell can't match a wildcard, it just passes the unmatched wildcard and other filenames:

*.o core

to the command (in this case, to rm) and lets the command decide what to do with it. So, with Bourne shell, what happens will depend on what your rm command does when it sees the literal characters *.o.

The Korn shell works like the Bourne shell.

You can make csh and tcsh act a lot like sh (and ksh) by setting the shell's nonomatch option. Without nonomatch set, the shell sees a nonmatching wildcard and never runs ls at all. Then I set nonomatch and the shell passes the unmatched wildcard on to ls, which prints its own error message:

% ls a*

ls: No match.

% set nonomatch

% ls a*

ls: a*: No such file or directory

In bash Version 1, the option allow_null_glob_expansion converts nonmatching wildcard patterns into the null string. Otherwise, the wildcard is left as is without expansion. Here's an example with echo (Section 27.5), which simply shows the arguments that it gets from the shell. In the directory where I'm running this example, there are no names starting with a, but there are two starting with

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Suppose you're giving a command like the one below (not necessarily rm — this applies to any Unix command):

% rm /somedir/otherdir/*

Let's say that matches 100 files. The rm command gets 100 complete pathnames from the shell: /somedir/otherdir/afile, /somedir/otherdir/bfile, and so on. For each of these files, the Unix kernel has to start at the root directory, then search the somedir and otherdir directories before it finds the file to remove.

That can make a significant difference, especially if your disk is already busy. It's better to cd to the directory first and run the rm from there. You can do it in a subshell (with parentheses) (Section 43.7) if you want to, so you won't have to cd back to where you started:

&& Section 35.14

% (cd /somedir/otherdir && rm *)

There's one more benefit to this second way: you're not as likely to get the error Arguments too long. (Another way to handle long command lines is with the xargs (Section 28.17) command.)

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Normally when you run grep (Section 13.1) on a group of files, the output lists the filename along with the line containing the search pattern. Sometimes you want to know only the names of the files, and you don't care to know the line (or lines) that match. In this case, use the -l (lowercase letter "l") option to list only filenames where matches occur. For example, the following command:

% grep -l R6 

               file1 file2 ... 

               > r6.filelist

searches the files for a line containing the string R6, produces a list of those filenames, and stores the list in r6.filelist. (This list might represent the files containing Release 6 documentation of a particular product.) Because these Release 6 files can now be referenced by one list, you can treat them as a single entity and run various commands on them all at once:

'...' Section 28.14

% lpr `cat r6.filelist`         

               Print only the Release 6 files

% grep UNIX `cat r6.filelist`   

               Search limited to the Release 5 files

You don't have to create a file list, though. You can insert the output of a grep directly into a command line with command substitution. For example, to edit only the subset of files containing R6, you would type:

% vi `grep -l R6 

               file1 file2 ...`

(Of course, you also could use a wildcard like file* instead of a list of filenames.)

grep -l is also good for shell programs that need to check whether a file contains a particular string. The traditional way to do that test is by throwing away grep's output and checking its exit status:

if grep something somefile >/dev/null

then ...

If somefile is huge, though, grep has to search all of it. Adding the grep -l option saves time because grep can stop searching after it finds the first matching line.

—DG and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

You can use the grep ( Section 13.2) option -c to tell you how many occurrences of a pattern appear in a given file, so you can also use it to find files that don't contain a pattern (i.e., zero occurrences of the pattern). This is a handy technique to package into a shell script.

Let's say you're indexing a DocBook (SGML) document and you want to make a list of files that don't yet contain indexing tags. What you need to find are files with zero occurrences of the string <indexterm>. (If your tags might be uppercase, you'll also want the -i option (Section 9.22).) The following command:

% grep -c "<indexterm>" chapter*

might produce the following output:

chapter1.sgm:10

chapter2.sgm:27

chapter3.sgm:19

chapter4.sgm:0

chapter5.sgm:39

   ...

This is all well and good, but suppose you need to check index entries in hundreds of reference pages. Well, just filter grep's output by piping it through another grep. The previous command can be modified as follows:

% grep -c "<indexterm>" chapter* | grep :0

This results in the following output:

chapter4.sgm:0

Using sed (Section 34.1) to truncate the :0, you can save the output as a list of files. For example, here's a trick for creating a list of files that don't contain index macros:

% grep -c "<indexterm>" * | sed -n 's/:0$//p' > ../not_indexed.list

The sed -n command prints only the lines that contain :0; it also strips the :0 from the output so that ../not_indexed.list contains a list of files, one per line. For a bit of extra safety, we've added a $ anchor (Section 32.5) to be sure sed matches only 0 at the end of a line — and not, say, in some bizarre filename that contains :0. (We've quoted (Section 27.12) the $ for safety — though it's not really necessary in most shells because $/ can't match shell variables.) The .. pathname (Section 1.16) puts the not_indexed.list file into the parent directory — this is one easy way to keep grep from searching that file, but it may not be worth the bother.

To edit all files that need index macros added, you could type this:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: nom

The nom (no match) script takes filenames (usually expanded by the shell) from its command line. It outputs all filenames in the current directory that don't match. As Section 33.2 shows, some shells have an operator — ! or ^ — that works like nom, but other shells don't. Here are some examples of nom:

To get the names of all files that don't end with .ms:
```
% nom *.ms

                  
```
To edit all files whose names don't have any lowercase letters, use command substitution (Section 27.14):
```
% vi `nom *[a-z]*`

                  
```
To copy all files to a subdirectory named Backup (except Backup itself):
```
% cp `nom Backup` Backup

                  
```

Here's the script:

trap Section 35.17, case Section 35.11, $* Section 35.20, comm Section 11.8

#! /bin/sh

temp=/tmp/NOM$$

stat=1     # Error exit status (set to 0 before normal exit)

trap 'rm -f $temp; exit $stat' 0 1 2 15



# Must have at least one argument, and all have to be in current directory:

case "$*" in

"") echo Usage: `basename $0` pattern 1>&2; exit ;;

*/*)    echo "`basename $0` quitting: I can't handle '/'s." 1>&2; exit ;;

esac



# ls gives sorted file list. -d=don't enter directories, -1=one name/line.

ls -d ${1+"$@"} > $temp   # Get filenames we don't want to match

ls -1 | comm -23 - $temp  # Compare to current dir; output names we want

stat=0

The -d option (Section 8.5) tells ls to list the names of any directories, not their contents. The ${1+"$@"} (Section 36.7) works around a problem in some Bourne shells. You can remove the -1 option on the script's ls command line if your version of ls lists one filename per line by default; almost all versions of ls do that when they're writing into a pipe. Note that nom doesn't know about files whose names begin with a dot (.); you can change that if you'd like by adding the ls -A option (uppercase letter "A", which isn't on all versions of

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

^H^H^H are ASCII backspace characters. Written printably in email and Usenet messages, they're a tongue-in-cheek way of "erasing" the characters before without actually erasing them. They let you say "I didn't want you to see that" when you actually do.

sed (stream editor) amazes me. Why? It's not just that sed can edit data as it streams through a pipe (like all well-behaved Unix filters (Section 1.5) do). sed can test and branch and substitute and hold and exchange data as it streams through, but so can almost any scripting language. Maybe it's the minimalist in me that loves a tiny program (by today's standards, at least) with just a few operations — but operations so well-chosen that they make the tool powerful for its size. Sure, sure, Perl probably can do everything that sed can — and do each of those things in twenty different ways. Ah, I've got it: when I'm trying to do anything more than a simple substitution on data streaming by, sed's elegant simplicity almost forces me to strip a problem to its basics, to think of what I really need to do. No functions, no libraries, nothing except beautifully simple functionality.

[As someone who learned Perl regular expressions before I learned sed, I can relate to what Jerry is saying. One of the things I like about the classic Unix toolbox programs like sed is that they really do force you into a sort of Shaker-like elegant simplicity; the best programs, no matter what the language, have a quality like a Shaker chair: pure function, but with a respect for the fact that function doesn't have to be ugly. — SJC]

End of sermon. ;-) Even if you aren't into elegance and simplicity, and you just wanna get the job done, what do we cover about sed that might be useful?

In this chapter, we start out with the basics: Section 34.2, Section 34.3, Section 34.4, Section 34.5, Section 34.6, and Section 34.7 show you how to get started, how to test your scripts, and how to structure more advanced scripts. Section 34.8 through Section 34.14 cover regular expressions and complex transformations. Section 34.15 through Section 34.24 deal with advanced topics such as multiline matching and deletions, tests, and exiting a script when you're done.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

^H^H^H are ASCII backspace characters. Written printably in email and Usenet messages, they're a tongue-in-cheek way of "erasing" the characters before without actually erasing them. They let you say "I didn't want you to see that" when you actually do.

sed (stream editor) amazes me. Why? It's not just that sed can edit data as it streams through a pipe (like all well-behaved Unix filters (Section 1.5) do). sed can test and branch and substitute and hold and exchange data as it streams through, but so can almost any scripting language. Maybe it's the minimalist in me that loves a tiny program (by today's standards, at least) with just a few operations — but operations so well-chosen that they make the tool powerful for its size. Sure, sure, Perl probably can do everything that sed can — and do each of those things in twenty different ways. Ah, I've got it: when I'm trying to do anything more than a simple substitution on data streaming by, sed's elegant simplicity almost forces me to strip a problem to its basics, to think of what I really need to do. No functions, no libraries, nothing except beautifully simple functionality.

[As someone who learned Perl regular expressions before I learned sed, I can relate to what Jerry is saying. One of the things I like about the classic Unix toolbox programs like sed is that they really do force you into a sort of Shaker-like elegant simplicity; the best programs, no matter what the language, have a quality like a Shaker chair: pure function, but with a respect for the fact that function doesn't have to be ugly. — SJC]

End of sermon. ;-) Even if you aren't into elegance and simplicity, and you just wanna get the job done, what do we cover about sed that might be useful?

In this chapter, we start out with the basics: Section 34.2, Section 34.3, Section 34.4, Section 34.5, Section 34.6, and Section 34.7 show you how to get started, how to test your scripts, and how to structure more advanced scripts. Section 34.8 through Section 34.14 cover regular expressions and complex transformations. Section 34.15 through Section 34.24 deal with advanced topics such as multiline matching and deletions, tests, and exiting a script when you're done.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you are already familiar with global edits in other editors like vi or ex, you know most of what you need to know to begin to use sed. There are two things, though, that make it very different:

It doesn't change the file it edits. It is just what its name says: a "stream editor" — designed to take a stream of data from standard input (Section 43.1) or a file, transform it, and pass it to standard output (Section 43.1). If you want to edit a file, you have to write a shell wrapper (Section 34.4) to capture standard output and write it back into your original file.
sed commands are implicitly global. In an editor like ex, the command:
```
s/old/new/
```
will change "old" to "new" only on the current line unless you use the global command or various addressing symbols to apply it to additional lines. In sed, exactly the opposite is true. A command like the one above will be applied to all lines in a file. Addressing symbols are used to limit the extent of the match. (However, like ex, only the first occurrence of a pattern on a given line will be changed unless the g flag is added to the end of the substitution command.)

If all you want to do is make simple substitutions, you're ready to go. If you want to do more than that, sed has some unique and powerful commands.

This chapter makes no attempt to cover everything there is to know about sed. For the most part, this chapter simply contains advice on working with sed and extended explanations of how to use some of its more difficult commands.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you were using sed on the fly, as a stream editor (Section 34.2), you might execute it as simply as this:

% somecommand

                | sed 's/old/new/' | 

               othercommand

Given filenames, sed will read them instead of standard input:

% sed 's/old/new/' myfile

A simple script can go right on the command line. If you want to execute more than one editing command, you can use the -e option:

% sed -e 's/old/new/' -e '/bad/d' myfile

Or you can use semicolons (;), which are a sed command separator:

% sed 's/old/new/; /bad/d' myfile

Or (especially useful in shell scripts (Section 1.8)) you can use the Bourne shell's ability to understand multiline commands:

sed '

s/old/new/

/bad/d' myfile

Or you can put your commands into a file and tell sed to read that file with the -f option:

% sed -f scriptfile myfile

There's only one other command-line option: -n . sed normally prints every line of its input (except those that have been deleted by the editing script). But there are times when you want only lines that your script has affected or that you explicitly ask for with the p command. In these cases, use -n to suppress the normal output.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

All but the simplest sed scripts are often invoked from a "shell wrapper," a shell script (Section 35.2) that invokes sed and also contains the editing commands that sed executes. A shell wrapper is an easy way to turn what could be a complex command line into a single-word command. The fact that sed is being used might be transparent to users of the command.

Two shell scripts that you should immediately arm yourself with are described here. Both use a shell for loop ( Section 35.21) to apply the same edits to any number of files. But the first just shows the changes, so you can make sure that your edits were made correctly. The second writes the edits back into the original file, making them permanent.

Go to http://examples.oreilly.com/upt3 for more information on: checksed

The shell script checksed automates the process of checking the edits that sed makes. It expects to find the script file, sedscr, in the current directory and applies these instructions to the input files named on the command line. The output is shown by a pager program; the default pager is more .

#! /bin/sh

script=sedscr



for file

do

        echo "********** < = $file    > = sed output **********"

        sed -f $script "$file" | diff "$file" -

done | ${PAGER-more}

For example:

$ cat sedscr

s/jpeek@ora\.com/jpeek@jpeek.com/g

$ checksed home.html new.html

********** < = home.html    > = sed output **********

102c102

< <a href="mailto:jpeek@ora.com">Email it</a> or use this form:

--

> <a href="mailto:jpeek@jpeek.com">Email it</a> or use this form:

124c124

< Page created by: <a href="mailto:jpeek@ora.com">jpeek@ora.com</a>>

--

> Page created by: <a href="mailto:jpeek@jpeek.com">jpeek@jpeek.com</a>

********** < = new.html    > = sed output **********

22c22

< <a href="mailto:jpeek@ora.com">Send comments</a> to me!

---

> <a href="mailto:jpeek@jpeek.com">Send comments</a> to me!

If you find that your script did not produce the results you expected, perfect the editing script and run

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A sed command can specify zero, one, or two addresses. An address can be a line number, a line addressing symbol, or a regular expression ( Section 32.4) that describes a pattern.

If no address is specified, the command is applied to each line.
If there is only one address, the command is applied to any line matching the address.
If two comma-separated addresses are specified, the command is performed on the first matching line and all succeeding lines up to and including a line matching the second address. This range may match multiple times throughout the input.
If an address is followed by an exclamation mark (!), the command is applied to all lines that do not match the address.

To illustrate how addressing works, let's look at examples using the delete command, d. A script consisting of simply the d command and no address:

produces no output since it deletes all lines.

When a line number is supplied as an address, the command affects only that line. For instance, the following example deletes only the first line:

1d

The line number refers to an internal line count maintained by sed. This counter is not reset for multiple input files. Thus, no matter how many files were specified as input, there is only one line 1 in the input stream.

Similarly, the input stream has only one last line. It can be specified using the addressing symbol, $. The following example deletes the last line of input:

$d

The $ symbol should not be confused with the $ used in regular expressions, where it means the end of the line.

When a regular expression is supplied as an address, the command affects only the lines matching that pattern. The regular expression must be enclosed by slashes (/). The following delete command:

/^$/d

deletes only blank lines. All other lines are passed through untouched.

If you supply two addresses, you specify a range of lines over which the command is executed. The following example shows how to delete all lines surrounded by a pair of XHTML tags, in this case, <ul> and </ul>, that mark the start and end of an unordered list:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Combining a series of edits in a script can have unexpected results. You might not think of the consequences one edit can have on another. New users typically think that sed applies an individual editing command to all lines of input before applying the next editing command. But the opposite is true. sed applies every editing command to the first input line before reading the second input line and applying the editing script to it. Because sed is always working with the latest version of the original line, any edit that is made changes the line for subsequent commands. sed doesn't retain the original. This means that a pattern that might have matched the original input line may no longer match the line after an edit has been made.

Let's look at an example that uses the substitute command. Suppose someone quickly wrote the following script to change pig to cow and cow to horse:

s/pig/cow/

s/cow/horse/

The first command would change pig to cow as expected. However, when the second command changed cow to horse on the same line, it also changed the cow that had been a pig. So, where the input file contained pigs and cows, the output file has only horses!

This mistake is simply a problem of the order of the commands in the script. Reversing the order of the commands — changing cow into horse before changing pig into cow — does the trick.

Another way to deal with this effect is to use a pattern you know won't be in the document except when you put it there, as a temporary placeholder. Either way, you know what the "document" looks like after each step in the program.

s/pig/cXXXoXXXw/

s/cow/horse/

s/cXXXoXXXw/cow/

Some sed commands change the flow through the script. For example, the N command (Section 34.16) reads another line into the pattern space without removing the current line, so you can test for patterns across multiple lines. Other commands tell sed to exit before reaching the bottom of the script or to go to a labeled command. sed also maintains a second temporary buffer called the hold space. You can copy the contents of the pattern space to the hold space and retrieve it later. The commands that make use of the hold space are discussed in Section 34.14 and other articles after it.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I find that when I begin to tackle a problem using sed, I do best if I make a mental list of all the things I want to do. When I begin coding, I write a script containing a single command that does one thing. I test that it works, then I add another command, repeating this cycle until I've done all that's obvious to do. I say what's obvious because my list is not always complete, and the cycle of implement-and-test often adds other items to the list. Another approach involves actually typing the list of tasks into a file, as comments, and then slowly replacing them with sed commands. If you're one of the rare but highly appreciated breed that actually documents their code, you can just leave the comments in the script or expand on them.

It may seem to be a rather tedious process to work this way, and indeed there are a number of scripts where it's fine to take a crack at writing the whole script in one pass and then begin testing it. However, the one-step-at-a-time method is highly recommended for beginners, because you isolate each command and get to easily see what is working and what is not. When you try to do several commands at once, you might find that when problems arise, you end up recreating the recommended process in reverse; that is, removing or commenting out commands one by one until you locate the problem.

— DD

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Whether in sed or vi, when using the substitution command, a delimiter is required to separate the search pattern from the replacement string. The delimiter can be any character except blank or a newline (vi seems to be more restrictive than sed, although vim is extremely flexible). However, the usual practice is to use the slash (/) as a delimiter (for example, s/ search / replacement /).

When either the search pattern or the replacement string contains a slash, it is easier to change the delimiter character than to escape the slash. Thus, if the pattern was attempting to match Unix pathnames, which contain slashes, you could choose another character, such as a colon, as the delimiter:

s:/usr/mail:/usr2/mail:

Note that the delimiter appears three times and is required after the replacement. Regardless of which delimiter you use, if it does appear in the search pattern or the replacement, put a backslash (\) before it to escape it.

If you don't know what characters the search pattern might have (in a shell program that handles any kind of input, for instance), the safest choice for the delimiter can be a control character.

You can use any delimiter for a pattern address (not just a slash). Put a backslash before the first delimiter. For example, to delete all lines containing /usr/mail, using a colon (:) as the delimiter:

\:/usr/mail:d

—DD and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The backslash (\) in the replacement string of the sed substitution command is generally used to escape other metacharacters, but it is also used to include a newline in a replacement string.

Given the following input line where each item is separated by a tab:

Column1     Column2    Column3     Column4

we can replace the second tab character on each line with a newline character:

2 Section 34.12

s/TAB/\

/2

Note that no spaces are permitted after the backslash. This script produces the following result:

Column1     Column2

Column3     Column4

Another example comes from the conversion of a file for troff to HTML. It converts the following line for troff:

.Ah "Major Heading"

to a similar line for HTML:

<h1>Major Heading</h1>

The twist in this problem is that the line needs to be preceded and followed by a blank line. It is an example of writing a multiline replacement string:

/^\.Ah/{

s/\.Ah */\

\

<h1>

s/"//g

s/$/<h1>\

/

}

The first substitute command replaces .Ah with two newlines and <h1>. Each backslash at the end of the line is necessary to escape the newline. The second substitution removes the quotation marks. The last command matches the end of line in the pattern space (not the embedded newlines) and adds a close h1 tag and a newline after it.

— DD

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As a metacharacter, the ampersand (& ) represents the extent of the pattern match, not the line that was matched. For instance, you might use it to match a word and surround it with troff requests. The following example surrounds a word with point-size requests:

s/UNIX/\\s-2&\\s0/g

Because backslashes are also replacement metacharacters, two backslashes are necessary to output a single backslash. The & in the replacement string refers to the string which was originally matched, UNIX. If the input line is:

on the UNIX Operating System.

the substitute command produces:

on the \s-2UNIX\s0 Operating System.

The ampersand is particularly useful when the regular expression matches variations of a word. It allows you to specify a variable replacement string that corresponds to what was actually matched. For instance, let's say that you wanted to surround with parentheses any cross reference to a numbered section in a document. In other words, any reference such as

See Section

1.4

or See Section 12.9 should appear in parentheses, as (See Section 12.9). A regular expression can match the different combination of numbers, so we use & in the replacement string and surround whatever was matched:

s/See Section [1-9][0-9]*\.[1-9][0-9]*/(&)/

The ampersand makes it possible to reference the entire match in the replacement string.

In the next example, the backslash is used to escape the ampersand, which appears literally in the replacement section:

s/ORA/O'Reilly \& Associates, Inc./g

It's easy to forget about the ampersand appearing literally in the replacement string. If we had not escaped it in this example, the output would have been

O'Reilly ORA Associates,

Inc

.

— DD

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In sed, the substitution command provides metacharacters to select any individual portion of a string that is matched and recall it in the replacement string. A pair of escaped parentheses are used in sed to enclose any part of a regular expression and save it for recall. Up to nine "saves" are permitted for a single line. \ n is used to recall the portion of the match that was saved, where n is a number from 1 to 9 referencing a particular "saved" string in order of use. (Section 32.13 has more information.)

For example, when converting a plain-text document into HTML, we could convert section numbers that appear in a cross-reference into an HTML hyperlink. The following expression is broken onto two lines for printing, but you should type all of it on one line:

s/\([sS]ee \)\(Section \)\([1-9][0-9]*\)\.\([1-9][0-9]*\)/

   \1<a href="#SEC-\3_\4">\2\3.\4<\/a>/

Four pairs of escaped parentheses are specified. String 1 captures the word see with an upper- or lowercase s. String 2 captures the section number (because this is a fixed string, it could have been simply retyped in the replacement string). String 3 captures the part of the section number before the decimal point, and String 4 captures the part of the section number after the decimal point. The replacement string recalls the first saved substring as \1. Next starts a link where the two parts of the section number, \3 and \4, are separated by an underscore (_) and have the string SEC- before them. Finally, the link text replays the section number again — this time with a decimal point between its parts. Note that although a dot (.) is special in the search pattern and has to be quoted with a backslash there, it's not special on the replacement side and can be typed literally. Here's the script run on a short test document, using checksed (Section 34.4):

% checksed testdoc

********** < = testdoc    > = sed output **********

8c8

< See Section 1.2 for details.

---

> See <a href="#SEC-1_2">Section 1.2</a> for details.

19c19

< Be sure to see Section 23.16!

---

> Be sure to see <a href="#SEC-23_16">Section 23.16</a>!

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the more unusual options of sed's substitution command is the numeric flag that allows you to point to one particular match when there are many possible matches on a particular line. It is used where a pattern repeats itself on a line and the replacement must be made for only one of those occurrences by position. For instance, a line, perhaps containing tbl input, might contain multiple tab characters. Let's say that there are three tabs per line, and you'd like to replace the second tab with >. The following substitute command would do it:

s/TAB/>/2

TAB represents an actual tab character, which is otherwise invisible on the screen. If the input is a one-line file such as the following:

Column1TABColumn2TABColumn3TABColumn4

the output produced by running the script on this file will be:

Column1TABColumn2>Column3TABColumn4

Note that without the numeric flag, the substitute command would replace only the first tab. (Therefore, 1 can be considered the default numeric flag.) The range of the allowed numeric value is from 1 to 512, though this may be implementation-dependent.

— DD

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The transform command (y) is useful for exchanging lowercase letters for uppercase letters on a line. Effectively, it performs a similar function to tr (Section 21.11). It replaces any character found in the first string with the equivalent character in the second string. The command:

y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/

will convert any lowercase letter into the corresponding uppercase letter. The following:

y/abcdefghijklmnopqrstuvwxyz/nopqrstuvwxyzabcdefghijklm/

would perform a rot13 transformation — a simple form of encryption in which each alphabetic character is replaced by the character halfway through the alphabet. (rot13 encryption is sometimes used to keep offensive (or comical) news postings (Section 1.21) from being read except by someone who really means to (such as if you have read the joke and now want to read the punch line). Encryption and decryption are automatically supported by most news readers, but it's fun to see how simple the encryption is. By the way, the command above handles only lowercase letters; if we'd shown uppercase as well, the command would have run past the margins!)

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The pattern space is a buffer that contains the current input line. There is also a set-aside buffer called the hold space . The contents of the pattern space can be copied to the hold space, and the contents of the hold space can be copied to the pattern space. A group of commands allows you to move data between the hold space and the pattern space. The hold space is used for temporary storage, and that's it. Individual commands can't address the hold space or alter its contents.

The most frequent use of the hold space is to have it retain a duplicate of the current input line while you change the original in the pattern space. [It's also used as a way to do the "move" and "copy" commands that most editors have — but which sed can't do directly because it's designed for editing a stream of input text line-by-line. — GU] The commands that affect the hold space are:

Hold	h	Copy contents of pattern space to hold space, replacing previous.
	H	Append newline, then append contents of pattern space, to hold space.
Get	g	Copy contents of hold space to pattern space, replacing previous.
	G	Append newline, then append contents of hold space, to pattern space.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The transform command, y (Section 34.13), acts on the entire contents of the pattern space. It is something of a chore to do a letter-by-letter transformation of a portion of the line, but it is possible (though convoluted) as the following example demonstrates. [The real importance of this example is probably not the use of the y command, but the use of the hold space to isolate and preserve part of the line. — TOR]

While working on a programming guide, we found that the names of statements were entered inconsistently. They needed to be uppercase, but some were lowercase while others had an initial capital letter. While the task was simple — to capitalize the name of the statement — there were nearly a hundred statements and it seemed a tedious project to write that many explicit substitutions of the form:

s/find the Match statement/find the MATCH statement/g

The transform command could do the lowercase-to-uppercase conversion, but it applies the conversion to the entire line. The hold space makes this task possible because we use it to store a copy of the input line while we isolate and convert the statement name in the pattern space. Look at the script first:

# capitalize statement names

/the .* statement/{

    h

    s/.*the \(.*\) statement.*/\1/

    y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/

    G

    s/\(.*\)\n\(.*the \).*\( statement.*\)/\2\1\3/

}

The address limits the procedure to lines that match

the .*

statement

. Let's look at what each command does:

h

The hold command copies the current input line into the hold space. Using the sample line find the Match statement, we'll show what the contents of the pattern space and hold space contain. After the h command, the pattern space and hold space are identical.

Pattern space	find the Match statement
Hold space

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most programs that use regular expressions (Section 32.4) are able to match a pattern only on a single line of input. This makes it difficult to find or change a phrase, for instance, because it can start near the end of one line and finish near the beginning of the next line. Other patterns might be significant only when repeated on multiple lines.

sed has the ability to load more than one line into the pattern space. This allows you to match (and change) patterns that extend over multiple lines. In this article, we show how to create a multiline pattern space and manipulate its contents.

The multiline Next command, N, creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The original contents of the pattern space and the new input line are separated by a newline. The embedded newline character can be matched in patterns by the escape sequence \n. In a multiline pattern space, only the metacharacter ^ matches the newline at the beginning of the pattern space, and $ matches the newline at the end. After the Next command is executed, control is then passed to subsequent commands in the script.

The Next command differs from the next command, n, which outputs the contents of the pattern space and then reads a new line of input. The next command does not create a multiline pattern space.

For our first example, let's suppose that we wanted to change "Owner and Operator Guide" to "Installation Guide", but we found that it appears in the file on two lines, splitting between Operator and Guide. For instance, here are a few lines of sample text:

Consult Section 3.1 in the Owner and Operator

Guide for a description of the tape drives

available on your system.

The following script looks for Operator at the end of a line, reads the next line of input, and then makes the replacement:

/Operator$/{

    N

    s/Owner and Operator\nGuide/Installation Guide/

}

In this example, we know where the two lines split and where to specify the embedded newline. When the script is run on the sample file, it produces the two lines of output, one of which combines the first and second lines and is too long to show here. This happens because the substitute command matches the embedded newline but does not replace it. Unfortunately, you cannot use

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The operations of sed's most difficult commands — hold (h or H), get (g or G), and exchange (x) — can be explained, somewhat fancifully, in terms of an extremely deliberate medieval scrivener or amanuensis toiling to make a copy of a manuscript. His work is bound by several spatial restrictions: the original manuscript is displayed in one room; the set of instructions for copying the manuscript are stored in a middle room; and the quill, ink, and folio are set up in yet another room. The original manuscript and the set of instructions are written in stone and cannot be moved about. The dutiful scrivener, being sounder of body than mind, is able to make a copy by going from room to room, working on only one line at a time. Entering the room where the original manuscript is, he removes from his robes a scrap of paper to take down the first line of the manuscript. Then he moves to the room containing the list of editing instructions. He reads each instruction to see if it applies to the single line he has scribbled down.

Each instruction, written in special notation, consists of two parts: a pattern and a procedure. The scrivener reads the first instruction and checks the pattern against his line. If there is no match, he doesn't have to worry about the procedure, so he goes to the next instruction. If he finds a match, the scrivener follows the action or actions specified in the procedure.

He makes the edit on his piece of paper before trying to match the pattern in the next instruction. Remember, the scrivener has to read through a series of instructions, and he reads all of them, not just the first instruction that matches the pattern. Because he makes his edits as he goes, he is always trying to match the latest version against the next pattern; he doesn't remember the original line.

When he gets to the bottom of the list of instructions, and has made any edits that were necessary on his piece of paper, he goes into the next room to copy out the line. (He doesn't need to be told to print out the line.) After that is done, he returns to the first room and takes down the next line on a new scrap of paper. When he goes to the second room, once again he reads every instruction from first to last before leaving.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[Section 13.9 introduced a script called cgrep , a general-purpose, grep-like program built with sed. It allows you to look for one or more words that appear on one line or across several lines. This article explains the sed tricks that are necessary to do this kind of thing. It gets into territory that is essential for any advanced applications of this obscure yet wonderful editor. Section 34.14 through Section 34.17 have background information. — JP]

Let's review the two examples from Section 13.9. The first command below finds all lines containing the word system in the file main.c and shows 10 additional lines of context above and below each match. The second command finds all occurrences of the word "awk" where it is followed by the word "perl" somewhere within the next 3 lines:

cgrep -10 system main.c

cgrep -3 "awk.*perl"

Now the script, followed by an explanation of how it works:

case Section 35.11, expr Section 36.21, shift Section 35.22, ${?} Section 36.7, \~..~ Section 34.8, "$@" Section 35.20

#!/bin/sh

#  cgrep - multiline context grep using sed

#  Usage: cgrep [-context] pattern [file...]



n=3

case $1 in -[1-9]*)

    n=`expr 1 - "$1"`

    shift

esac

re=${1?}; shift



sed -n "

    1b start

    : top

    \~$re~{

        h; n; p; H; g

        b endif

    }

        N

        : start

        //{ =; p; }

    : endif

    $n,\$D

    b top

" "$@"

The sed script is embedded in a bare-bones shell wrapper ( Section 35.19) to parse out the initial arguments because, unlike awk and perl, sed cannot directly access command-line parameters. If the first argument looks like a -context option, variable n is reset to one more than the number of lines specified, using a little trick — the argument is treated as a negative number and subtracted from 1. The pattern argument is then stored in $re, with the ${1?} syntax causing the shell to abort with an error message if no pattern was given. Any remaining arguments are passed as filenames to the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The sed delete command, d, deletes the contents of the pattern space ( Section 34.14) and causes a new line of input to be read, with editing resuming at the top of the script. The Delete command, D, works slightly differently: it deletes a portion of the pattern space, up to the first embedded newline. It does not cause a new line of input to be read; instead, it returns to the top of the script, applying these instructions to what remains in the pattern space. We can see the difference by writing a script that looks for a series of blank lines and outputs a single blank line. The version below uses the delete command:

# reduce multiple blank lines to one; version using d command

/^$/{

   N

   /^\n$/d

}

When a blank line is encountered, the next line is appended to the pattern space. Then we try to match the embedded newline. Note that the positional metacharacters, ^ and $, match the beginning and the end of the pattern space, respectively. Here's a test file:

This line is followed by 1 blank line.



This line is followed by 2 blank lines.





This line is followed by 3 blank lines.







This line is followed by 4 blank lines.









This is the end.

Running the script on the test file produces the following result:

% sed -f sed.blank test.blank

This line is followed by 1 blank line.



This line is followed by 2 blank lines.

This line is followed by 3 blank lines.



This line is followed by 4 blank lines.

This is the end.

Where there was an even number of blank lines, all the blank lines were removed. Only when there was an odd number of blank lines was a single blank line preserved. That is because the delete command clears the entire pattern space. Once the first blank line is encountered, the next line is read in, and both are deleted. If a third blank line is encountered, and the next line is not blank, the delete command is not applied, and thus a blank line is output. If we use the multiline Delete command, /^\n$/D, we get a different result, and the one that we wanted.

The reason the multiline Delete command gets the job done is that when we encounter two blank lines, the Delete command removes only the first of the two. The next time through the script, the blank line will cause another line to be read into the pattern space. If that line is not blank, both lines are output, thus ensuring that a single blank line will be output. In other words, when there are two blank lines in the pattern space, only the first is deleted. When a blank line is followed by text, the pattern space is output normally.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are two ways in sed to avoid specified portions of a document while making the edits everywhere else. You can use the ! command to specify that the edit applies only to lines that do not match the pattern. Another approach is to use the b (branch) command to skip over portions of the editing script. Let's look at an example.

We've used sed to preprocess the input to troff so that double dashes ( -- ) are converted automatically to em-dashes ( — ) and straight quotes are converted to curly quotes. However, program examples in technical books are usually shown in a constant-width font that clearly shows each character as it appears on the computer screen. When typesetting a document, we don't want sed to apply the same editing rules within these examples as it does to the rest of the document. For instance, straight quotes should not be replaced by curly quotes.

Because program examples are set off by a pair of macros (something like .ES and .EE, for "Example Start" and "Example End"), we can use those as the basis for exclusion. Here's some sample text that includes an example:

.LP

The \fItrue\fP command returns a zero exit status.

As Joe says, "this is only useful in programming":

.ES

% \fBtrue\fP

% \fBecho "the status was $status"\fP

the status was 0

.EE

So you can say:

/^\.ES/,/^\.EE/!{

   s/^"/``/

       ...

   s/\\(em"/\\(em``/g

}

All of the commands enclosed in braces ({} ) will be subject to the initial pattern address.

There is another way to accomplish the same thing. The b command allows you to transfer control to another line in the script that is marked with an optional label. Using this feature, you could write the previous script like this:

/^\.ES/,/^\.EE/bend

s/^"/``/

   ...

s/\\(em"/\\(em``/g

:end

A label consists of a colon (:) followed by up to seven characters. If the label is missing, the b command branches to the end of the script. (In the example above, the label end was included just to show how to use one, but a label is not really necessary here.)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The test command, t, branches to a label (or the end of the script) if a successful substitution has been made on the currently addressed line. It implies a conditional branch. Its syntax is as follows:

[address]t[label]

If no label is supplied, control falls through to the end of the script. If label is supplied, execution resumes at the line following the label.

Let's look at a spelling corrector written by Greg Ubben. The script fixes common (in this example, silly) spelling goofs; the t command tells about corrections that were made:

h

s/seperate/separate/g

s/compooter/computer/g

s/said editor/sed editor/g

s/lable/label/g

t changed

b

: changed

p

g

s/.*/[WAS: &]/

t

First, h (Section 34.14) holds a copy of the current input line. Then, if any of the four substitutions succeed, the command

t

changed

branches to the corresponding label (

:

changed

) at the end of the script. Otherwise, if no s succeeded, the b command restarts the script on the next line (as always in sed, the input line is printed before the script restarts).

After the label, the script prints the current input line (the line with a spelling error — which, by now, has been corrected). Then g (Section 34.14) gets the original uncorrected line. An s command brackets that line [WAS: xxx ]. Here's some sample output:

$ sed -f sedscr afile

This is a separate test.

[WAS: This is a seperate test.]

I put a label on my computer!

[WAS: I put a lable on my compooter!]

That's all for now.

The final t in the script is a work-around for a bug in some versions of sed. Greg says, "The t flag is supposed to be reset after either the t command is executed or a new line of input is read, but some versions of sed don't reset it on a new line of input. So I added a do-nothing t to make sure it's reset after the previous always-true s///." Try the script without the extra t; if adding it makes the script work right, your sed has the bug and you might try a new version, like GNU sed.

—JP and DD

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The quit command, q, causes sed to stop reading new input lines (and stop sending them to the output). Its syntax is:

[line-address]q

Section 34.23

It can take only a single-line address. Once the line matching address (line-address) is reached, the script will be terminated.

For instance, the following one-liner uses the quit command to print the first ten lines from a file:

% sed '10q' myfile

   ...

sed prints each line until it gets to line 10 and quits.

The previous version is much more efficient than its functional equivalent:

-n Section 34.3

% sed -n '1,10p' myfile

(especially if myfile is a long file) because sed doesn't need to keep reading its input once the patterns in the script are satisfied.

One possible use of q is to quit a script after you've extracted what you want from a file. There is some inefficiency in continuing to scan through a large file after sed has found what it is looking for.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The sed quit command, q (Section 34.22), is very useful for getting sed to stop processing any more input once you've done what you want.

However, you need to be very careful not to use q in any sed script that writes its edits back to the original file. After q is executed, no further output is produced. It should not be used in any case where you want to edit the front of the file and pass the remainder through unchanged. Using q in this case is a dangerous beginner's mistake.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Feeding sed (Section 34.1) newlines is easy; the real trick is getting them past the C shell and its derivatives (tcsh has the same problem, on the systems where we've tested it).

The sed documentation says that in order to insert newlines in substitute commands, you should quote them with backslashes. [Surround the commands with single quotes ('), as Chris has. If you use double quotes ("), this script will become s/foo/bar/ because of the way quoting works with backslashes and newlines ( Section 27.12). — JP]:

sed -e 's/foo/b\

a\

r/'

Indeed, this works quite well in the Bourne shell and derivatives, such as bash, which do what I consider the proper thing (Section 27.12) with this input. The C shell, however, thinks it is smarter than you are and removes the trailing backslashes (Section 27.13), and instead you must type:

sed -e 's/foo/b\\

a\\

r/'

Probably the best solution is to place your sed commands in a separate file (Section 34.3) to keep the shell's sticky fingers off them.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A shell script need be no more than a command line saved in a file. For example, let's assume that you'd like a compact list of all the users who are currently logged in on the system.

A command like this might do the trick:

% who | cut -c1-8 | sort -u | pr -l1 -8 -w78 -t

A list of logged-in users should come out in columns, looking something like this:

abraham  appleton biscuit  charlie  charlott fizzie   howard   howie

hstern   jerry    kosmo    linda    ocshner  peterson root     ross

sutton   yuppie

We used four Unix commands joined with pipes:

who (Section 2.8) gives a list of all users.
cut -c1-8 (Section 21.14) outputs columns 1-8 of the who output — the usernames.
sort -u (Section 22.6) puts names in order and takes out names of users who are logged on more than once.
pr -l1 -8 -w78 -t (Section 21.15, Section 45.6) takes the list of usernames, one per line, and makes it into 8 columns on 78-character-wide lines. (The -l1 is the lowercase letter L followed by the digit 1.)

If you wanted to do this frequently, wouldn't it be better if all you had to do was type something like:

% loggedin

to get the same result? Here's how:

Start a text editor on a new file named loggedin.
If your system supports the special #! notation (Section 36.2) (and it probably does), the first line of the script file should be:
```
#!/bin/sh
```
Otherwise, leave the first line blank. (When the first line of a script is blank, most shells will start a Bourne shell to read it. Section 36.2 has more information.)
I think that the second line of a shell script should always be a comment to explain what the script does. (Use more than one line, if you want.) A comment starts with a hash mark (#); all characters after it on the line are ignored. Oh, and try to make sure there's a bit of whitespace between the comment character and the actual comment; that's a pet peeve of mine:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A shell script need be no more than a command line saved in a file. For example, let's assume that you'd like a compact list of all the users who are currently logged in on the system.

A command like this might do the trick:

% who | cut -c1-8 | sort -u | pr -l1 -8 -w78 -t

A list of logged-in users should come out in columns, looking something like this:

abraham  appleton biscuit  charlie  charlott fizzie   howard   howie

hstern   jerry    kosmo    linda    ocshner  peterson root     ross

sutton   yuppie

We used four Unix commands joined with pipes:

who (Section 2.8) gives a list of all users.
cut -c1-8 (Section 21.14) outputs columns 1-8 of the who output — the usernames.
sort -u (Section 22.6) puts names in order and takes out names of users who are logged on more than once.
pr -l1 -8 -w78 -t (Section 21.15, Section 45.6) takes the list of usernames, one per line, and makes it into 8 columns on 78-character-wide lines. (The -l1 is the lowercase letter L followed by the digit 1.)

If you wanted to do this frequently, wouldn't it be better if all you had to do was type something like:

% loggedin

to get the same result? Here's how:

Start a text editor on a new file named loggedin.
If your system supports the special #! notation (Section 36.2) (and it probably does), the first line of the script file should be:
```
#!/bin/sh
```
Otherwise, leave the first line blank. (When the first line of a script is blank, most shells will start a Bourne shell to read it. Section 36.2 has more information.)
I think that the second line of a shell script should always be a comment to explain what the script does. (Use more than one line, if you want.) A comment starts with a hash mark (#); all characters after it on the line are ignored. Oh, and try to make sure there's a bit of whitespace between the comment character and the actual comment; that's a pet peeve of mine:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the great things about Unix is that it's made up of individual utilities, "building blocks" like cat and grep, that you run from a shell prompt. Using pipes, redirection, filters, and so on, you can combine those utilities to do an incredible number of things. Shell programming lets you take the same commands you'd type at a shell prompt and put them into a file you can run by just typing its name. You can make new programs that combine Unix programs (and other shell scripts) in your own way to do exactly what you need. If you don't like the way a program works, you can write a shell script to do just what you want.

Because many Unix users use the shell every day, they don't need to learn a whole new language for programming, just some tips and techniques. In fact, this chapter covers a lot of programming techniques that you'll want to use even when you aren't programming. For example, loops and tests are handy on the command line.

(This series of articles does assume that you've written programs in some language before or are generally familiar with programming concepts. If you haven't and aren't, you might start with a more comprehensive shell programming book.)

Unix has plenty of other scripting languages — Perl, Python, and Tcl/Tk are some of the best known. So when should you write a script with the shell and when shouldn't you? That's a personal choice; as you learn more languages and their strengths and weaknesses, you're better able to choose the best one for a situation. My rule of thumb is something like this. I write a shell script if:

It's a script I developed at the command line, so it's easy to just drop those same commands into a file.
I know some Unix utility that'll do just what I want.
It has to be portable to a system that might not have another scripting language I'd rather use.
The (possibly) slower speed of forking processes to run Unix utilities (especially in loops) doesn't matter.
The script simply has to make a few decisions — like whether standard input is a

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many Unix utilities, including the shell, need information about you and what you're doing in order to do a reasonable job.

What kinds of information? Well, to start with, a lot of programs (particularly editors) need to know what kind of terminal you're using. The shell needs to know where any commands you want to use are likely to be found. Lots of Unix programs (like mail programs) include a command to start an editor as a subprocess; they like to know your favorite editor. And so on.

Of course, one could always write programs that made you put all this information on the command line. For example, you might have to type commands like:

% mail -editor vi -term aardvark48 -favoritecolor blue_no_red

But your favorite editor probably doesn't change every day. (Nor will your favorite color.) The terminal you use may change frequently, but it certainly won't change from the time you log in until the time you log out. And you certainly wouldn't want to type something like this whenever you want to send mail.

Rather than forcing you to type this information with every command, Unix uses environment variables to store information you'd rather not worry about. For example, the TERM ( Section 5.2) environment variable tells programs what kind of terminal you're using. Any programs that care about your terminal type know (or ought to know) that they can read this variable, find out your terminal type, and act accordingly.

Similarly, the directories that store the commands you want to execute are listed in the PATH (Section 35.6) variable. When you type a command, your shell looks through each directory in your PATH variable to find that command. Presumably, Unix wouldn't need a PATH variable if all commands were located in the same directory, but you'll soon be writing your own commands (if you aren't already), and storing them in your own "private" command directories (Section 7.4), and you'll need to tell the shell how to find them (Section 27.6).

Environment variables are managed by your shell. The difference between environment variables and regular

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

No, this is not about the psychology of computing. It's just a reminder of one important point.

In the environment variable overview (Section 35.3) we said that each process gets its own copy of its parent's environment variables. We chose those words carefully, and if you think about them, you won't make one common mistake.

Sooner or later, almost everyone writes a shell script that gathers some information, sets a few environment variables, and quits. The writer then wonders why there's no trace of the "new" environment variables to be found. The problem is simple. A Unix process (Section 24.3) cannot change its parent's environment; a Unix process gets its own copy of the parent's environment, and any changes it makes it keeps to itself. A process can make changes and pass them to its children, but there's no way of going in reverse.

(You can't teach an old dog new tricks.)

[This is important in window systems, too. Environment variables set in one window (more exactly, in one process) probably won't affect any process running in any other window. To affect all windows, set the environment variable before you start the window system. For instance, if you log in and then type startx from a shell prompt to start X, you can set environment variables from that prompt or from that shell's setup files (Section 3.3). — JP]

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We've said that environment variables are used to store information that you'd rather not worry about, and that there are a number of standard environment variables that many Unix programs use. These are often called "predefined" environment variables — not because their values are predefined, but because their names and uses are predefined. Here are some important ones:

PATH (Section 35.6): contains your command search path (Section 27.6). This is a list of directories in which the shell looks to find commands. It's usually set in one of your shell setup files (Section 3.3).
EDITOR and VISUAL: can be loaded with the name of your favorite editor. They're usually set in one of your shell setup files. Some programs distinguish between EDITOR (usually set to a line editor (Section 20.1) such as ed) and VISUAL (set to a full-screen editor like vi). Many people don't follow that convention; they set both to the same editor. (The Korn shell checks VISUAL and EDITOR, in that order, to determine your command editing mode (Section 30.14).)
PRINTER (Section 45.4) or LPDEST: can be loaded with the name of your default printer. This is quite useful at a site with many printers — you don't need to tell lpr or lp (Section 45.2) which printer to use. (PRINTER works on systems that print with lpr , and LPDEST is for lp.) This variable is usually set in one of your shell setup files.
PWD: may contain the absolute pathname of your current directory. It's set automatically by the cd command in some Unix shells. PWD may be fooled by cding through symbolic links.
HOME (Section 31.11) (called LOGDIR on some systems)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Of all the environment variables, the PATH and TERM variables are the most important. The others are often great conveniences, but PATH and TERM can make your life miserable if they get screwed up.

The PATH variable is just a list of directories separated by colon (:) characters. The shell searches through these directories in order whenever it needs to find a command. So, if you want to execute commands in /bin, /usr/bin, /usr/local, the current directory, and your personal bin directory, you would put a line like the one below in your .login file. An empty entry (: as the first or last character, or :: in the middle) means "the current directory."

$HOME/bin Section 7.4

setenv PATH /bin:/usr/bin:/usr/local::$HOME/bin

Section 27.6 explains more about setting the path.

The most common problem with PATH is that, somehow, it gets deleted. This usually happens if you try to change PATH and do so incorrectly. When PATH is deleted, your shell can find only its built-in commands (Section 1.9) and commands for which you give the complete pathname. Here's a demonstration:

% setenv PATH   

               Set PATH to null accidentally

% ls

ls: Command not found.

Needless to say, this can be very frustrating — especially if you can't figure out what's going on. There are a couple of easy fixes. The easiest is just to log out and log back in again. (logout is a built-in C shell command, so you won't have trouble finding it. If you get an error message like "Not login shell," try exit instead.) Another fix is to read (Section 35.29) whichever initialization file defined your PATH variable, usually .login for C shell users or .profile or .bash_profile for Bourne or bash shell users, respectively:

% source ~/.login

$ . $HOME/.profile

bash$ . $HOME/.bash_profile

This will almost certainly give you some of your path back; the problem is that a lot of initialization files merely add a few "private" directories to a system-wide default path. In this case, just execute the system-wide initialization files first (if your system has them). Their pathnames vary:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

For csh and zsh, it's slightly incorrect to say that PATH contains the search list for commands. It's a bit more complicated. The PATH environment variable is used to set the path shell variable; that is, whenever you use setenv PATH (Section 35.6) in csh or export PATH in zsh, the shell modifies path accordingly. For example:

setenv PATH /bin:/usr/bin:/usr/local::$HOME/bin   csh

export PATH=/bin:/usr/bin:/usr/local::$HOME/bin   zsh

In PATH, an empty entry (::) stands for the current directory. The shells' path shell variable (Section 35.9) is the actual search list. Its syntax is slightly different; the list of directories is enclosed in parentheses ([XREF: UPT-ART-0508]), and the directories are separated by spaces. For example:

~ Section 30.11

set path=(/bin /usr/bin /usr/local . ~/bin)   csh



path=(/bin /usr/bin /usr/local . ~/bin)       zsh

If you set the path shell variable, the shell will automatically set the PATH environment variable. You don't need to set both. Many people set the shell variable instead of the environment variable.

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The most important environment variable for X Window System clients is DISPLAY. When a user logs in at an X terminal, the DISPLAY environment variable in each xterm window is set to her X terminal's hostname followed by :0.0.

ruby:joan % echo $DISPLAY

ncd15.ora.com:0.0

When the same user logs in at the console of the workstation named sapphire that's running X, the DISPLAY environment variable is defined as just :0.0:

sapphire:joan % echo $DISPLAY

:0.0

The DISPLAY environment variable is used by all X clients to determine what X server to display on. Since any X client can connect to any X server that allows it, all X clients need to know what display to connect to upon startup. If DISPLAY is not properly set, the client cannot execute:

sapphire:joan % setenv DISPLAY foo:0

sapphire:joan % xterm

xterm Xt error: Can't open display:

You can override the value of DISPLAY by using the -display command-line option. For example:

sapphire:joan % xterm -display sapphire:0.0 &

The first part of the display name (up to and including the colon) identifies the type of connection to use and the host that the server is running on. The second part (in most cases, the string 0.0) identifies a server number and an optional screen number. In most cases, the server and screen numbers will both be 0. You can omit the screen number name if the default (screen 0) is correct.

Note that we used both :0.0 and sapphire:0.0 to access the local console display of the workstation named sapphire. Although both these names will work, they imply different ways of connecting to the X server.

The : character without an initial hostname specifies that the client should connect using UNIX domain sockets (IPC).
Since processes can communicate via IPC only if they are running on the same host, you can use a leading colon or the unix keyword in a display name only if both the client and server are running on the same host — that is, for local clients displaying to the local console display of a workstation.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Shell variables are really just the "general case" of environment variables (Section 35.3). If you're a programmer, remember that a Unix shell really runs an interpreted programming language. Shell variables belong to the shell; you can set them, print them, and work with them much as you can in a C program (or a FORTRAN program or a BASIC program). If you're not a programmer, just remember that shell variables are pigeonholes that store information for you or your shell to use.

If you've read the articles on environment variables, you realize that we defined them in exactly the same way. How are shell variables different from environment variables? Whenever you start a new shell or a Unix program, it inherits all of its parent's environment variables. However, it does not inherit any shell variables; it starts with a clean slate (except, possibly, variables in some shell setup files (Section 3.3)). If you're a programmer, you can think of environment variables as "global" variables, while shell variables are "local" variables. By convention, shell variables have lowercase names.

Just as some programs use certain environment variables, the shell expects to use certain shell variables. For example, the C shell uses the history (Section 30.1) variable to determine how many of your previous commands to remember; if the noclobber (Section 43.6) variable is defined, the C shell prevents you from damaging files by making mistakes with standard output. Most users insert code into their .cshrc or .tcshrc (Section 3.3) files to define these important variables appropriately. Alternatively, they split them up into context-specific files and then read them into their environment (Section 35.29) as needed.

To set a shell variable, use one of these commands:

% set 

               name

               =

               value   

               C shell

$ name

               =

               value   

                 other shells

As a special case, if you omit

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Each time you type a command line at a shell prompt, you can see what happens and decide what command to run next. But a shell script needs to make decisions like that itself. A case statement helps the script make decisions. A case statement compares a string (usually taken from a shell or environment variable (Section 35.9, Section 35.3)) to one or more patterns. The patterns can be simple strings (words, digits, etc.) or they can be case wildcard expressions (Section 35.11). When the case statement finds a pattern that matches the string, it executes one or more commands.

Here's an example that tests your TERM ( Section 5.2) environment variable. If you're using a vt100 or tk4023 terminal, it runs a command to send some characters to your terminal. If you aren't on either of those, it prints an error and quits:

exit Section 35.16

+case "$TERM" in

vt100 echo 'ea[w' | tr 'eaw' '\033\001\027' ;;

tk4023)  echo "*[p23" ;;

*)  # Not a VT100 or tk4023.  Print error message:

    echo "progname: quitting: you aren't on a VT100 or tk4023." 1>&2

    exit

    ;;

esac

Here are more details about how this works. The statement compares the string between the words case and in to the strings at the left-hand edge of the lines ending with a ) (right parenthesis) character. If it matches the first case (in this example, if it's the vt100), the command up to the ;; is executed. The ;; means "jump to the esac" (esac is "case" spelled backwards). You can put as many commands as you want before each ;;, but put each command on a separate line (or separate commands on a line with semicolons (Section 28.16)).

If the first pattern doesn't match, the shell tries the next case — here, tk4023. As above, a match runs the command and jumps to the esac. No match? The next pattern is the wildcard *. It matches any answer other than vt100 or tk4023 (such as xterm or an empty string).

You can use as many patterns as you want to. The first one that matches is used. It's okay if none of them match. The style doesn't matter much. Pick one that's readable and be consistent.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A case statement ( Section 35.10) is good at string pattern matching. Its "wildcard" pattern-matching metacharacters work like the filename wildcards (Section 1.13) in the shell, with a few twists. Here are some examples:

?): Matches a string with exactly one character like a, 3, !, and so on.
?*): Matches a string with one or more characters (a nonempty string).
[yY]|[yY][eE][sS]): Matches y, Y or yes, YES, YeS, etc. The | means "or."
/*/*[0-9]): Matches a file pathname, like /xxx/yyy/somedir/file2, that starts with a slash, contains at least one more slash, and ends with a digit.
'What now?'): Matches the pattern What now?. The quotes (Section 27.12) tell the shell to treat the string literally: not to break it at the space and not to treat the ? as a wildcard.
"$msgs"): Matches the contents of the msgs variable. The double quotes let the shell substitute the variable's value; the quotes also protect spaces and other special characters from the shell. For example, if msgs contains first next, this would match the same string, first next.

To clarify: in bash, for example, the case statement uses the same pathname expansion rules it uses elsewhere in the shell, to determine how to expand the value. In other shells, such as ksh, there are minor differences (such as a relaxation of special treatment for . and / characters). See the manual page for your shell if you have any questions or concerns about what rules your shell will follow.

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When a Unix process (command) runs, it can return a numeric status value to the parent process that called (started) it. The status can tell the calling process whether the command succeeded or failed. Many (but not all) Unix commands return a status of zero if everything was okay and nonzero (1, 2, etc.) if something went wrong. A few commands, such as grep and diff, return a different nonzero status for different kinds of problems; see your online manual pages (or just experiment!) to find out.

The Bourne shell puts the exit status of the previous command in the question mark (?) variable. You can get its value by preceding it with a dollar sign ($), just like any other shell variable. For example, when cp copies a file, it sets the status to 0. If something goes wrong, cp sets the status to 1:

$ cp afile /tmp

$ echo $?

0

$ cp afiel /tmp

cp: afiel: No such file or directory

$ echo $?

1

In the C shell, use the status variable instead (tcsh supports both):

% cp afiel /tmp

cp: afiel: No such file or directory

% echo $status

1

tcsh> cp afiel /tmp

cp: afiel: No such file or directory

tcsh> echo $status

1

Of course, you usually don't have to display the exit status in this way, because there are several ways (Section 35.13, Section 35.14, Section 35.15) to use the exit status of one command as a condition of further execution.

Go to http://examples.oreilly.com/upt3 for more information on: true, false

Two simple Unix utilities do nothing but return an exit status. true returns a status of 0 (zero); false returns 1 (one). There are GNU versions on the web site — and no, they don't have any amazing extra features. ;-)

bash and zsh have a handy way to reverse the status of a command line: put an exclamation point (!) before it. Let's look at a simple example (of course, you'd use ! with something besides

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you are going to write a shell script of any complexity at all, you need some way to write "conditional expressions." Conditional expressions are nothing more than statements that have a value of "true" or "false", such as "Am I wearing pants today?" or "Is it before 5 p.m.?" or "Does the file indata exist?" or "Is the value of $aardvark greater than 60?"

The Unix shell is a complete programming language. Therefore, it allows you to write "if" statements with conditional expressions — just like C, BASIC, Pascal, or any other language. Conditional expressions can also be used in several other situations, but most obviously, they're the basis for any sort of if statement. Here's the syntax of an if statement for the Bourne shell:

+if conditional

then

    # do this if conditional returns a zero ("true") status

    one-or-more-commands

else

    # do this if conditional returns non-zero ("false") status

    one-or-more-commands

fi

Depending on how many different ways the command might exit, and thus the varying values its exit status may have, you may want to use either a case statement or elif (for testing multiple conditionals in a single if/else block.)

You can omit the else and the block of code following it. However, you can't omit the then or the fi. If you want to omit the then (i.e., if you want to do something special when condition is false, but nothing when it is true), write the statement like this:

if conditional

then

    :    # do nothing

else

    # do this if conditional returns non-zero ("false") status

    one-or-more-commands

fi

Note that this uses a special null command, a colon (:) (Section 36.6). There's another, more useful way of expressing the inverse of a condition (do something if conditional is not "true"), the || operator (Section 35.14) (two vertical bars). You can use this to write an

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The shells let you test for success right on the command line. This gives you a very efficient way to write quick and comprehensible shell scripts.

I'm referring to the || and && operators and in particular, the || operator. comm1 || comm2 is typically explained as "execute the command on the right if the command on the left failed." I prefer to explain it as an "either-or" construct: "execute either comm1 or comm2." While this isn't really precise, let's see what it means in context:

cat filea fileb > filec || exit

This means "either cat the files or exit." If you can't cat the files (if cat returns an exit status of 1), you exit (Section 24.4). If you can cat the files, you don't exit. You execute the left side or the right side.

I'm stretching normal terminology a bit here, but I think it's necessary to clarify the purpose of ||. By the way, we could give the poor user an error message before flaming out (which, by the way, is a way to write an "inverse if (Section 35.13)):

cat filea fileb > filec || {

   echo sorry, no dice 1>&2

   exit 1

}

Similarly, comm1 && comm2 means "execute comm1 AND comm2," or execute comm2 if comm1 succeeds. (But if you can't execute the first, don't do any.) This might be helpful if you want to print a temporary file and delete it immediately.

lpr file && rm file

If lpr fails for some reason, you want to leave the file around. Again, I want to stress how to read this: print the file and delete it. (Implicitly: if you don't print it, don't delete it.)

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Bourne shell has two kinds of loops that run a command and test its exit status. An until loop will continue until the command returns a zero status. A while loop will continue while the command returns a zero status.

The until loop runs a command repeatedly until it succeeds. That is, if the command returns a nonzero status, the shell executes the body of the loop and then runs the loop control command again. The shell keeps running the command until it returns a zero status, as shown in the following example:

% cat sysmgr

#!/bin/sh

until who | grep "^barb "

do sleep 60

done

echo The system manager just logged on.

% sysmgr 

                  &

[1] 2345

     ...time passes...

barb     ttyp7   Jul 15 09:30

The system manager just logged on.

The loop runs who (Section 2.8) and pipes that output to grep (Section 13.1), which searches for any line starting with barb and a space. (The space makes sure that usernames like barbara don't match.) If grep returns a nonzero status (no lines matched), the shell waits 60 seconds. Then the loop repeats, and the script tries the who | grep command again. It keeps doing this until grep returns a zero status — then the loop is broken and control goes past the done line. The echo command prints a message and the script quits. (I ran this script in the background so I could do something else while I waited for Barb.)

This is also a useful way to get someone with whom you share a machine to turn on their cell phone: just set a loop to wait until they login and then send them a write message (in case they don't always check their email, like a few nocturnal system administrators I know).

[A Bourne shell until loop is not identical to the until construction in most programming languages, because the condition is evaluated at the top of the loop. Virtually all languages with an until loop evaluate the condition at the bottom. — ML]

Go to http://examples.oreilly.com/upt3 for more information on: catsaway

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most standard Unix toolbox commands return a status (Section 35.12). Your shell script should, too. This section shows how to set the right exit status for both normal exits and error exits.

To end a shell script and set its exit status, use the exit command. Give exit the exit status that your script should have. If it has no explicit status, it will exit with the status of the last command run.

Go to http://examples.oreilly.com/upt3 for more information on: bkedit

Here's an example, a rewrite of the bkedit script from Section 35.13. If the script can make a backup copy, the editor is run and the script returns the exit status from vi (usually 0). If something goes wrong with the copy, the script prints an error and returns an exit status of 1. Here's the script:

#!/bin/sh

if cp "$1" "$1.bak"

then

    vi "$1"

    exit   # Use status from vi

else

    echo "bkedit quitting: can't make backup?" 1>&2

    exit 1

fi

Here's what happens if I run it without a filename:

$ bkedit

cp: usage: cp fn1 fn2 or cp fn1 [fn2...] dir

bkedit quitting: can't make backup?

And here's what's left in the exit status variable:

$ echo $?

1

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're running a shell script and you press your interrupt key (Section 5.8) (like CTRL-c), the shell quits right away. That can be a problem if you use temporary files in your script, because the sudden exit might leave the temporary files there. The trap command lets you tell the shell what to do before it exits. A trap can be used for a normal exit, too. See Table 35-1.

Table 35-1: Some Unix signal numbers for trap commands
Signal number	Signal name	Explanation
0	EXIT	exit command
1	HUP	When session disconnected
2	INT	Interrupt — often CTRL-c
3	QUIT	Quit — often CTRL-\
9	KILL	Kill, often used as a way to stop an errant program (it cannot be caught, so don't bother to trap it)
15	TERM	From kill command

Here's a script named zmore that uses a temporary file named /tmp/zmore$$ in a system temporary-file directory. The shell will replace $$

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Bourne shell read command reads a line of one or more words from the keyboard (or standard input) and stores the words in one or more shell variables. This is usually what you use to read an answer from the keyboard. For example:

echo -n "Type the filename: "

read filename

Here is how the read command works:

If you give the name of one shell variable, read stores everything from the line into that variable:
```
read varname

                  
```
If you name more than one variable, the first word typed goes into the first variable, the second word into the second variable, and so on. All leftover words go into the last variable. For example, with these commands:
```
echo -n "Enter first and last name: "

read fn ln
```
if a user types John Smith, the word John would be available from $fn and Smith would be in $ln. If the user types Jane de Boes, then Jane would be in $fn and the two words de Boes are in $ln.

Some Bourne shells have a built-in function named line that reads a line from standard input and writes it to standard output. Use it with command substitutions (Section 28.14):

value=`line`

Go to http://examples.oreilly.com/upt3 for more information on: grabchars

The grabchars program lets you read from the keyboard without needing to press RETURN.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Although most scripts for most languages can execute directly (Section 36.3) without needing the Bourne shell, it's common to "wrap" other scripts in a shell script to take advantage of the shell's strengths. For instance, sed can't accept arbitrary text on its command line, only commands and filenames. So you can let the shell handle the command line (Section 35.20) and pass information to sed via shell variables, command substitution, and so on. Simply use correct quoting (Section 27.12) to pass information from the shell into the "wrapped" sed script:

|| Section 35.14

#!/bin/sh

# seder - cd to directory in first command-line argument ($1),

# read all files and substitute $2 with $3, write result to stdout

cd "$1" || exit

sed "s/$2/$3/g" *

In SunExpert magazine, in his article on awk (January, 1991), Peter Collinson suggests a stylization similar to this for awk programs in shell scripts (Section 35.2):

#!/bin/sh

awkprog='

/foo/{print $3}

/bar/{print $4}'



awk "$awkprog" $*

He argues that this is more intelligible in long pipelines because it separates the program from the command. For example:

grep foo $input | sed .... | awk "$awkprog" - | ...

Not everyone is thrilled by the "advantages" of writing awk this way, but it's true that there are disadvantages to writing awk the standard way.

Here's an even more complex variation:

<<\ Section 27.16

#!/bin/sh

temp=/tmp/awk.prog.$$

cat > $temp <<\END

/foo/{print $3}

/bar/{print $4}

END

awk -f $temp $1

rm -f $temp

This version makes it a bit easier to create complex programs dynamically. The final awk command becomes the equivalent of a shell eval (Section 27.8); it executes something that has been built up at runtime. The first strategy (program in shell variable) could also be massaged to work this way.

As another example, a program that I used once was really just one long pipeline, about 200 lines long. Huge

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

To write flexible shell scripts, you usually want to give them command-line arguments. As you've seen in other articles (Section 35.16, Section 35.17), $1 holds the first command-line argument. The Bourne shell can give you arguments through the ninth, $9. The Korn shell and some other newer Bourne-type shells understand ${10} for the tenth argument, and so on.

If you've been reading this series (Section 35.2) of articles in order, you saw the zmore (Section 35.17) script that accepted just one command-line argument. If you put "$@" in a script, the shell will replace that string with a quoted (Section 27.12) set of the script's command-line arguments. Then you can pass as many arguments as you want, including pathnames with unusual characters (Section 14.11):

% zmore report memo "savearts/What's next?"

The third argument has a perfectly legal filename; we see more and more of them on our system — especially filesystems that are networked to computers like the Macintosh, or on systems that use windowing systems to run graphical programs such as FrameMaker, where spaces and other "special" characters in filenames are common. Double-quoting all arguments through the script helps to be sure that the script can handle these unusual but legal pathnames.

In this case, we want the arguments to be passed to the GNU zcat command. Let's change the zmore script to read:

zcat "$@" >$temp

When the shell runs the script with the arguments shown above, the command line will become:

zcat "report" "memo" "savearts/What's next?" >/tmp/zmore12345

On some Bourne shells, if there are no command-line arguments, the "$@" becomes a single empty argument (Section 37.5), as if you'd typed this:

zcat "" >/tmp/zmore12345

In this case, the zcat command would complain that it can't find a file. (Of course, in this script, the case would prevent this problem. But not all scripts test the number of arguments.)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sometimes you want a script that will step through the command-line arguments one by one. (The "$@" parameter (Section 35.20) gives you all of them at once.) The Bourne shell for loop can do this. The for loop looks like this:

for arg in list

do

    ...handle $arg...

done

If you omit the in list, the loop steps through the command-line arguments. It puts the first command-line argument in arg (or whatever else you choose to call the shell variable (Section 35.9)), then executes the commands from do to done. Then it puts the next command-line argument in arg, does the loop, and so on, ending the loop after handling all the arguments.

For an example of a for loop, let's hack on the zmore (Section 35.17) script.

case Section 35.11

#!/bin/sh

# zmore - Uncompress file(s), display with "more"

# Usage: zmore [more options] file [...files]

stat=1  # Default exit status; reset to 0 before normal exit

temp=/tmp/zmore$$

trap 'rm -f $temp; exit $stat' 0

trap 'echo "`basename $0`: Ouch! Quitting early..." 1>&2' 1 2 15



files=  switches=

for arg

do

    case "$arg" in

    -*) switches="$switches $arg" ;;

    *)  files="$files $arg" ;;

    esac

done



case "$files" in

"") echo "Usage: `basename $0` [more options] file [files]" 1>&2 ;;

*)  for file in $files

    do

        zcat "$file" | more $switches

    done

    stat=0

    ;;

esac

We added a for loop to get and check each command-line argument. For example, let's say that a user typed the following:

% zmore -s afile ../bfile

The first pass through the for loop, $arg is -s. Because the argument starts with a minus sign (-), the case treats it as an option. Now the switches variable is replaced by its previous contents (an empty string), a space, and -s. Control goes to the esac and the loop repeats with the next argument.

The next argument, afile, doesn't look like an option. So now the files variable will contain a space and afile.

The loop starts over once more with ../bfile in

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A for loop (Section 35.21) is great if you want to handle all of the command-line arguments to a script, one by one. But, as is often the case, some arguments are options that have their own arguments. For example, in the command grep -f filename, filename is an argument to -f; the option and its argument need to be processed together. One good way to handle this is with a combination of while (Section 35.15), test (Section 35.26), case (Section 35.10), and shift. Here's the basic construct:

while [ $# -gt 0 ]

do

    case "$1" in

        -a) options="$options $1";;

            ...

        -f) options="$options $1"

            argfile="$2"

            shift

            ;;

         *) files="$files $1";;

    esac

    shift

done

The trick is this: shift removes an argument from the script's argument list, shifting all the others over by one ($1 disappears, $2 becomes $1, $3 becomes $2, and so on). To handle an option with its own argument, do another shift. The while loop uses test (Section 35.26) to check that $# — the number of arguments — is greater than zero and keeps going until this is no longer true, which only happens when they have all been used up.

Meanwhile, all the case has to do is to test $1 against the desired option strings. In the simple example shown above, we simply assume that anything beginning with a minus sign is an option, which we (presumably) want to pass on to some program that is being invoked by the script. So all we do is build up a shell variable that will eventually contain all the options. It would be quite possible to do something else instead, perhaps setting other shell variables or executing commands.

We assume that anything without a minus sign is a file. This last case could be written more robustly with a test to be sure the argument is a file. Here's an example of a simple script that uses this construct to pass an option and some files to pr and from there to a program that converts text to PostScript and on to the print spooler (or you could convert SGML or XML files to PDF, whatever):

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Normally a for loop (Section 35.21) iterates until it has processed all its word arguments. while and until loops (Section 35.15) iterate until the loop control command returns a certain status. But sometimes — for instance, if there's an error — you want a loop to immediately terminate or jump to the next iteration. That's where you use break and continue, respectively.

break terminates the loop and takes control to the line after done. continue skips the rest of the commands in the loop body and starts the next iteration. Here's an example of both. An outer loop is stepping through a list of directories. If we can't cd to one of them, we'll abort the loop with break. The inner loop steps through all entries in the directory. If one of the entries isn't a file or isn't readable, we skip it and try the next one.

'...' Section 28.14, || Section 35.14, * Section 1.13, test Section 35.26

for dir in `find $HOME/projdir -type d -print`

do

    cd "$dir" || break

    echo "Processing $dir"



    for file in *

    do

        test -f "$file" -a -r "$file" || continue

            ...process $dir/$file...

    done

done

With nested loops (like the file loop above), which loop is broken or continued? It's the loop being processed at that time. So the continue here restarts the inner (file) loop. The break terminates the outer (directory) loop, which means the inner loop is also terminated. Note also that the -print argument to find is often redundant in the absence of another expression, depending on your version of find.

Here we've used break and continue within for loops, after the shell's || operator. But you can use them anywhere within the body of any loop — in an if statement within a while loop, for instance.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most shell scripts need to handle command-line arguments — options, filenames, and so on. Section 35.20, Section 35.21, and Section 35.22 show how to parse command lines with any Bourne shell. Those methods have two problems. You can't combine arguments with a single dash, e.g., -abc instead of -a -b -c. You also can't specify arguments to options without a space in between, e.g., -b arg in addition to -b arg.

Your Bourne shell may have a built-in command named getopts. getopts lets you deal with multiple complex options without these constraints. To find out whether your shell has getopts, see your online sh or getopts(1) manual page.

getopt takes two or more arguments. The first is a string that can contain letters and colons (:). Each letter names a valid option; if a letter is followed by a colon, the option requires an argument. The second and following arguments are the original command-line options; you'll usually give "$@" (Section 35.20) to pass all the arguments to getopt.

getopt picks each option off the command line, checks to see if the option is valid, and writes the correct option to its standard output. If an option has an argument, getopt writes the argument after its option. When getopt finds the first nonoption argument (the first argument that doesn't start with a - character), it outputs two dashes (--) and the rest of the arguments. If getopt finds an invalid option, or an option that should have an argument but doesn't, it prints an error message and returns a nonzero status (Section 35.12).

Go to http://examples.oreilly.com/upt3 for more information on: opttest

Your script can use a loop to parse the getopt output. Here's an example script named opttest that shows how getopt works:

|| Section 35.14, : Section 36.6

#!/bin/sh

set -- `getopt "ab:" "$@"` || {

    echo "Usage: `basename $0` [-a] [-b name] [files]" 1>&2

    exit 1

}

echo "Before loop, command line has: $*"

aflag=0  name=NONE

while :

do

    case "$1" in

    -a) aflag=1 ;;

    -b) shift; name="$1" ;;

    --) break ;;

    esac

    shift

done

shift    # REMOVE THE TRAILING --

echo "aflag=$aflag / name=$name / Files are $*"

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

[Most of this article, except IFS and --, also applies to the C shell. — JP]

You can pass options and arguments to a shell as you start it, as in:

sh -v file1 file2

and also when a script is invoked with #!. The set command lets you set command-line parameters, including most shell options, after you've started the shell. This simple idea has more uses than you might realize.

The Bourne shell command line can have options like -e (exit if any command returns nonzero status). It can also have other arguments; these are passed to shell scripts. You can set new command-line parameters while you're typing interactive commands (at a shell prompt) or in a shell script.

To reset the command-line parameters, just type set followed by the new parameters. For example, to ask the shell to show expanded versions of command lines after you type them, set the -v (verbose) option (Section 27.15):

$ set -v

$ mail $group1 < message

mail andy ellen heather steve wilma < message

$ mail $group2 < message

mail jpeek@jpeek.com randy@xyz.edu yori@mongo.medfly.com < message

$ set +v

On many Bourne shells, typing set +v cancels the v option. On other (mostly early) shells, there's no + feature. To work around that problem, you could start a subshell (Section 24.4) with sh -v, run the commands there, then exit the subshell to cancel the verbose option.

Go to http://examples.oreilly.com/upt3 for more information on: users

You can put filenames or any other strings in the command-line parameters interactively or from a shell script. That's handy for storing and parsing the output of a Unix command with backquotes (Section 28.14). For example, you can get a list of all logged-in users from the parameters $1, $2, and so on. Use users (or rusers to find all the logged in users on the local network) if your system has it. Otherwise, use who ( Section 2.8) and cut (Section 21.14) to strip off everything but the usernames:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix has a command called test that does a lot of useful tests. For instance, test can check whether a file is writable before your script tries to write to it. It can treat the string in a shell variable as a number and do comparisons ("Is that number less than 1000?"). You can combine tests, too ("If the file exists and it's readable and the message number is more than 500..."). Some versions of test have more tests than others. For a complete list, read your shell's manual page (if your shell has test built in (Section 1.9)) or the online test(1) manual page.

The test command returns a zero status (Section 35.12) if the test was true and a nonzero status otherwise, so people usually use test with if , while, or until. Here's a way your program could check to see if the user has a readable file named .signature in the home directory:

$HOME Section 35.5, $myname Section 35.28

if test -r $HOME/.signature

then

    ...Do whatever...

else

    echo "$myname: Can't read your '.signature'.  Quitting." 1>&2

    exit 1

fi

The test command also lets you test for something that isn't true. Add an exclamation point (!) before the condition you're testing. For example, the following test is true if the .signature file is not readable:

if test ! -r $HOME/.signature

then

    echo "$myname: Can't read your '.signature'.  Quitting." 1>&2

    exit 1

fi

Unix also has a version of test (a link to the same program, actually) named [. Yes, that's a left bracket. You can use it interchangeably with the test command with one exception: there has to be a matching right bracket (]) at the end of the test. The second example above could be rewritten this way:

if [ ! -r $HOME/.signature ]

then

    echo "$myname: Can't read your '.signature'.  Quitting." 1>&2

    exit 1

fi

Be sure to leave space between the brackets and other text. There are a couple of other common gotchas caused by empty arguments; Section 37.3 and Section 37.4 have workarounds.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you write a new program or shell script, you'll probably want to be sure that its name doesn't conflict with any other commands on the system. For instance, you might wonder whether there's a command named tscan. You can check by typing one of the commands in the following example. If you get output (besides an error) from one of them, there's probably already a command with the same name. (The type command works on ksh, bash, and many Bourne shells; I've shown it with a dollar sign ($) prompt.)

which Section 2.6, whereis Section 2.3, alias Section 29.2

% man 1 tscan

No manual entry for tscan in section 1.

% which tscan

no tscan in . /xxx/ehuser/bin /usr/bin/X11 /usr/local/bin ...

% whereis tscan

tscan:

% alias tscan

%

% whatis tscan

tscan:

$ type tscan

tscan not found

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A Unix program should use its name as the first word in error messages it prints. That's important when the program is running in the background or as part of a pipeline — you need to know which program has the problem:


               someprog: quitting: can't read file xxxxxx

It's tempting to use just the program name in the echo commands:

echo "someprog: quitting: can't read file $file" 1>&2

If you ever change the program name, however, it's easy to forget to fix the messages. A better way is to store the program name in a shell variable at the top of the script file and use the variable in all messages:

myname=someprog

   ...

echo "$myname: quitting: can't read file $file" 1>&2

Even better, use the $0 parameter. The shell automatically puts the script's name there. But $0 can have the absolute pathname of the script, such as /xxx/yyy/bin/someprog. The basename (Section 36.13) program fixes this: basename strips off the head of a pathname — everything but the filename.

For example, if $0 is /u/ehuser/bin/sendit:

myname="`basename $0`"

would put sendit into the myname shell variable.

Just as you can make links (Section 10.3) to give Unix files several names, you can use links to give your program several names (Section 36.8). For instance, see the script named ll, lf, lg (...and so on). Use $0 to get the current name of the program.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As Section 35.4 explains, Unix programs can never, ever modify the environment of their parents. A program can only modify the environment that later will be passed to its children. This is a common mistake that many new Unix users make: they try to write a program that changes a directory (or does something else involving an environment variable) and attempt to figure out why it doesn't work. You can't do this. If you write a program that executes the cd command, that cd will be effective within your program — but when the program finishes, you'll be back in your original (parent) shell.

One workaround is to "source" the shell script file (for csh and bash) or run it as a "dot" script (sh, ksh, and bash, too). For example, if the file named change-my-directory contains:

cd /somewhere/else

you could use the following commands to change the current directory of the current shell:

% source change-my-directory

$ . change-my-directory

The source and . commands read a script file into the current shell instead of starting a child shell. These commands work only for shell script files (files containing command lines as you'd type them at a shell prompt). You can't use source or . to read a binary (directly executable) file into the shell.

If your shell doesn't have shell functions (Section 29.11), you can simulate them (Section 29.14) with the . command. It acts a lot like a subroutine or function in a programming language.

—ML and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

So far, we have discussed some shell function basics (Section 29.11), using examples such as the

mx(

)

function that uses sed and dig to find out what host accepts mail for a given address. In that example, we simply made a set of complex functionality available as a quick and easy (and short) call from the command line. But you can also define functions and use them within shell scripts, or you can use the . and source commands to include those functions from an external file (Section 35.29).

We've also discussed using functions to automate repetitive tasks (Section 29.11), such as calculating factorials.

For now, let's demonstrate both of these approaches specifically with respect to defining a function to automate a repetitive task and then sharing the function with other shell scripts. Using the

mx(

)

function we defined earlier, let's put it into its own file, mx.sh, and store it in our personal shell function library directory (in this case, $HOME/lib):

$ cat > ~/lib/mx.sh

function mx( ) {

# Look up mail exchanger for host(s)

for host

do

    echo "==== $host ===="

    dig "$host" mx in |

    sed -n '/^;; ANSWER SECTION:/,/^$/{

            s/^[^;].* //p

    }'

done

}

^D

$ more !$

function mx( ) {

# Look up mail exchanger for host(s)

for host

do

    echo "==== $host ===="

    dig "$host" mx in |

    sed -n '/^;; ANSWER SECTION:/,/^$/{

            s/^[^;].* //p

    }'

done

}

$

Now the file ~/lib/mx.sh contains a function named mx( ) — fair enough, but let's say we want to be able to pass a list of hosts (determined dynamically on a regular basis, say, from spam-fighting tools that find open SMTP proxies) to a shell script, and have that shell script email a form letter to the postmaster address at that host. We will call the shell script proxynotify, and call it as follows:

$ proxynotify < proxyList

proxylist contains a list of hosts, one per line, in the com domain. We want to iterate over them and mail the postmaster for the domain, rather than mailing directly to an address such as

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This chapter has a bunch of tricks and techniques for programming with the Bourne shell. Some of them are documented but hard to find; others aren't documented at all. Here is a summary of this chapter's articles:

The first group of articles is about making a file directly executable with #! on the first line. On many versions of Unix, an executable file can start with a first line like this:
```
#!/path/to/interpreter

                  
```
The kernel will start the program named in that line and give it the file to read. Chris Torek's Usenet classic, Section 36.2, explains how #! started. Section 36.3 explains that your "shell scripts" may not need a shell at all.
The next bunch of articles are about processes and commands. The exec command, Section 36.5, replaces the shell with another process; it can also be used to change input/output redirection (see below). The : (colon) operator evaluates its arguments and returns a zero status — Section 36.6 explains why you should care.
Next are techniques for handling variables and parameters. Parameter substitution, explained in Section 36.7, is a compact way to test, set, and give default values for variables. You can use the $0 parameter and Unix links to give the same script multiple names and make it do multiple things; see Section 36.8. Section 36.9 shows the easy way to get the last command-line argument. Section 36.10 has an easy way to remove all the command-line arguments.
Four articles cover sh loops. A for loop usually reads a list of single arguments into a single shell variable. Section 36.11 shows how to make the for loop read from standard input. Section 36.12 has techniques for making a for loop set more than one variable. The dirname and basename commands can be used to split pathnames with a loop; see Section 36.13. A while loop can have more than one command line at the start; see Section 36.14.
Next is an assortment of articles about input/output. Section 36.15 introduces open files and file descriptors — there's more to know about standard input/output/error than you might have realized! Section 36.16 has a look at file-descriptor handling in the Bourne shell, swapping standard output and standard error.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This chapter has a bunch of tricks and techniques for programming with the Bourne shell. Some of them are documented but hard to find; others aren't documented at all. Here is a summary of this chapter's articles:

The first group of articles is about making a file directly executable with #! on the first line. On many versions of Unix, an executable file can start with a first line like this:
```
#!/path/to/interpreter

                  
```
The kernel will start the program named in that line and give it the file to read. Chris Torek's Usenet classic, Section 36.2, explains how #! started. Section 36.3 explains that your "shell scripts" may not need a shell at all.
The next bunch of articles are about processes and commands. The exec command, Section 36.5, replaces the shell with another process; it can also be used to change input/output redirection (see below). The : (colon) operator evaluates its arguments and returns a zero status — Section 36.6 explains why you should care.
Next are techniques for handling variables and parameters. Parameter substitution, explained in Section 36.7, is a compact way to test, set, and give default values for variables. You can use the $0 parameter and Unix links to give the same script multiple names and make it do multiple things; see Section 36.8. Section 36.9 shows the easy way to get the last command-line argument. Section 36.10 has an easy way to remove all the command-line arguments.
Four articles cover sh loops. A for loop usually reads a list of single arguments into a single shell variable. Section 36.11 shows how to make the for loop read from standard input. Section 36.12 has techniques for making a for loop set more than one variable. The dirname and basename commands can be used to split pathnames with a loop; see Section 36.13. A while loop can have more than one command line at the start; see Section 36.14.
Next is an assortment of articles about input/output. Section 36.15 introduces open files and file descriptors — there's more to know about standard input/output/error than you might have realized! Section 36.16 has a look at file-descriptor handling in the Bourne shell, swapping standard output and standard error.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Once upon a time, there was the Bourne shell. Since there was only "the" shell, there was no trouble deciding how to run a script: run it with the shell. It worked, and everyone was happy.

Along came progress and wrote another shell. The people thought this was good, for now they could choose their own shell. So some chose the one, and some the other, and they wrote shell scripts and were happy. But one day someone who used the "other" shell ran a script by someone who used the "other other" shell, and alas! it bombed spectacularly. The people wailed and called upon their Guru for help.

"Well," said the Guru, "I see the problem. The one shell and the other are not compatible. We need to make sure that the shells know which other shell to use to run each script. And lo! the one shell has a `comment' called :, and the other a true comment called #. I hereby decree that henceforth, the one shell will run scripts that start with :, and the other those that start with #." And it was so, and the people were happy.

But progress was not finished. This time he noticed that only shells ran scripts and thought that if the kernel too could run scripts, this would be good, and the people would be happy. So he wrote more code, and now the kernel could run scripts but only if they began with the magic incantation #! , and if they told the kernel which shell ran the script. And it was so, and the people were confused.

For the #! looked like a "comment." Though the kernel could see the #! and run a shell, it would not do so unless certain magic bits were set. And if the incantation were mispronounced, that too could stop the kernel, which, after all, was not omniscient. And so the people wailed, but alas! the Guru did not respond. And so it was, and still it is today. Anyway, you will get best results from a 4BSD machine by using

#! /bin/sh

or:

#! /bin/csh

as the first line of your script. #!

/bin/csh

-f

is also helpful on occasion, and it's usually faster because

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If your Unix understands files that start with:

#!/interpreter/program

(and nearly all of them do by now) you don't have to use those lines to start a shell, such as #!/bin/sh. If your script is just starting a program like awk, Unix can start the program directly and save execution time. This is especially useful on small or overloaded computers, or when your script has to be called over and over (such as in a loop).

First, here are two scripts. Both scripts print the second word from each line of text files. One uses a shell; the other runs awk directly:

% cat with_sh

#!/bin/sh

awk '

{ print $2 }

' $*

% cat no_sh

#!/usr/bin/awk -f

{ print $2 }

% cat afile

one two three four five

Let's run both commands and time ( Section 26.2) them. (This is running on a very slow machine. On faster systems, this difference may be harder to measure — though the difference can still add up over time.)

% time with_sh afile

two

0.1u 0.2s 0:00 26%

% time no_sh afile

two

0.0u 0.1s 0:00 13%

One of the things that's really important to understand here is that when the kernel runs the program on the interpreter line, it is given the script's filename as an argument. If the intepreter program understands a file directly, like /bin/sh does, nothing special needs to be done. But a program like awk or sed requires the -f option to read its script from a file. This leads to the seemingly odd syntax in the example above, with a call to awk -f with no following filename. The script itself is the input file!

One implication of this usage is that the interpreter program needs to understand # as a comment, or the first interpreter-selection line itself will be acted upon (and probably rejected by) the interpreter. (Fortunately, the shells, Perl, sed, and awk, among other programs, do recognize this comment character.)

[One last comment: if you have GNU time or some other version that has a verbose mode, you can see that the major difference between the two invocations is in terms of the page faults each requires. On a relatively speedy Pentium III/450 running RedHat Linux, the version using a shell as the interpreter required more than twice the major page faults and more than three times as many minor page faults as the version calling

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As Section 36.3 explains, you can use #! /path/name to run a script with the interpreter located at /path/name in the filesystem. The problem comes if a new version of the interpreter is installed somewhere else or if you run the script on another system that has a different location. It's usually not a problem for Bourne shell programmers: /bin/sh exists on every Unix-type system I've seen. But some newer shells — and interpreters like Perl — may be lurking almost anywhere (although this is becoming more and more standardized as Perl and other tools like it become part of standard Linux distributions and the like). If the interpreter isn't found, you'll probably get a cryptic message like scriptname: Command not found, where scriptname is the name of the script file.

The env command will search your PATH (Section 35.6) for an interpreter, then execute (exec (Section 24.2), replace itself) with the interpreter. If you want to try this, type env ls; env will find and run ls for you. This is pretty useless when you have a shell around to interpret your commands — because the shell can do the same thing without getting env involved. But when the kernel interprets an executable file that starts with #!, there's no shell (yet!). That's where you can use env. For instance, to run your script with zsh, you could start its file with:

#!/usr/bin/env zsh

   ...zsh script here...

The kernel execs /usr/bin/env, then env finds and execs the zsh it found. Nice trick, eh? What do you think the problem is? (You have ten seconds... tick, tick, tick...) The catch is: if the env command isn't in /usr/bin on your system, this trick won't work. So it's not as portable as it might be, but it's still handy and probably still better than trying to specify the pathname of a less common interpreter like zsh.

Running an interpreter this way can also be a security problem. Someone's PATH might be wrong; for instance, it might execute some random command named

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The exec command executes a command in place of the current shell; that is, it terminates the current shell and starts a new process (Section 24.3) in its place.

Historically, exec was often used to execute the last command of a shell script. This would kill the shell slightly earlier; otherwise, the shell would wait until the last command was finished. This practice saved a process and some memory. (Aren't you glad you're using a modern system? This sort of conservation usually isn't necessary any longer unless your system limits the number of processes each user can have.)

exec can be used to replace one shell with another shell:

% exec ksh

$

without incurring the additional overhead of having an unused shell waiting for the new shell to finish.

exec also manipulates file descriptors (Section 36.16) in the Bourne shell. When you use exec to manage file descriptors, it does not replace the current process. For example, the following command makes the standard input of all commands come from the file formfile instead of the default place (usually, your terminal):

exec < formfile

—ML and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some people think that the Bourne shell's : is a comment character. It isn't, really. It evaluates its arguments and returns a zero exit status (Section 35.12). Here are a few places to use it:

Replace the Unix true command to make an endless while loop (Section 35.15). This is more efficient because the shell doesn't have to start a new process each time around the loop (as it does when you use while true):
```
while :

do

   commands

done
```
(Of course, one of the commands will probably be break, to end the loop eventually. This presumes that it is actually a savings to have the break test inside the loop body rather than at the top, but it may well be clearer under certain circumstances to do it that way.)
When you want to use the else in an if ( Section 35.13) but leave the then empty, the : makes a nice "do-nothing" place filler:
```
if something

then :

else

   commands

fi
```
If your Bourne shell doesn't have a true # comment character (but nearly all of them do nowadays), you can use : to "fake it." It's safest to use quotes so the shell won't try to interpret characters like > or | in your "comment":
```
: 'read answer and branch if < 3 or > 6'
```
Finally, it's useful with parameter substitution (Section 35.7) like ${ var ?} or ${ var = default }. For instance, using this line in your script will print an error and exit if either the USER or HOME variables aren't set:
```
: ${USER?} ${HOME?}
```

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Bourne shell has a handy set of operators for testing and setting shell variables. They're listed in Table 36-1.

Table 36-1: Bourne shell parameter substitution operators
Operator	Explanation
`${` var `:-` default `}`	If var is not set or is empty, use default instead.
`${` var `:=` default `}`	If var is not set or is empty, set it to default and use that value.
`${` var `:+` instead `}`	If var is set and is not empty, use instead. Otherwise, use nothing (null string).
`${` var `:?` message `}`	If var is set and is not empty, use its value. Otherwise, print message, if any, and exit from the shell. If message is missing, print a default message (which depends on your shell).

If you omit the colon (:) from the expressions in Table 36-1, the shell doesn't check for an empty parameter. In other words, the substitution happens whenever the parameter is set. (That's how some early Bourne shells work: they don't understand a colon in parameter substitution.)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're writing:

several programs that do the same kinds of things,
programs that use a lot of the same code (as you're writing the second, third, etc., programs, you copy a lot of lines from the first program), or
a program with several options that make big changes in the way it works,

you might want to write just one program and make links (Section 10.4, Section 10.3) to it instead. The program can find the name you called it with and, through case or test commands, work in different ways. For instance, the Berkeley Unix commands ex, vi, view, edit, and others are all links to the same executable file. This takes less disk space and makes maintenance easier. It's usually sensible only when most of the code is the same in each program. If the program is full of name tests and lots of separate code, this technique may be more trouble than it's worth.

Depending on how the script program is called, this name can be a simple relative pathname like prog or ./prog — it can also be an absolute pathname like /usr/joe/bin/prog (Section 31.2 explains pathnames). There are a couple of ways to handle this in a shell script. If there's just one main piece of code in the script, as in the lf script, a case that tests $0 might be best. The asterisk (*) wildcard at the start of each case (see Section 35.11) handles the different pathnames that might be used to call the script:

case "$0" in

*name1)

    ...do this when called as name1...

    ;;

*name2)

    ...do this when called as name2...

    ;;

    ...

*)  ...print error and exit if $0 doesn't match...

    ;;

esac

You might also want to use basename (Section 36.13) to strip off any leading pathname and store the cleaned-up $0 in a variable called myname. You can test $myname anywhere in the script and also use it for error messages:

myname=`basename $0`

   ...

case "$myname" in

   ...



echo "$myname: aborting; error in xxxxxx" 1>&2

   ...

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Do you need to pick up the last parameter $1, $2 ... from the parameter list on the command line? It looks like eval \$$# would do it:

eval Section 27.8

$ set foo bar baz

$ eval echo \$$#

baz

except for a small problem with sh argument syntax:

$ set m n o p q r s t u v w x

$ echo $11

m1

$11 means ${1}1, not ${11}. Trying ${11} directly gives bad substitution. (More recent shells, such as bash, do support the ${11} syntax, however, to arbitrary lengths. Our copy of bash, for example, allowed at least 10240 command line arguments to set with recall of the last via ${10240}). Your mileage may vary.

The only reliable way to get at the last parameter in the Bourne shell is to use something like this:

for i do last="$i"; done

The for loop assigns each parameter to the shell variable named last; after the loop ends, $last will have the last parameter. Also, note that you won't need this trick on all sh-like shells. The Korn shell, zsh, and bash understand ${11}.

— CT

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The shift (Section 35.22) command "shifts away" one command-line parameter. You can shift three times if there are three command-line parameters. Many shells also can take an argument, like shift 3, that tells how many times to shift; on those shells, you can shift $# (Section 35.20) to unset all parameters.

The portable way to unset all command-line parameters is probably to set (Section 35.25) a single dummy parameter, then shift it away:

+set x

shift

Setting the single parameter wipes out whatever other parameters were set before.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

An obvious place to use a Bourne shell for loop (Section 35.21) is to step through a list of arguments — from the command line or a variable. But combine the loop with backquotes (Section 28.14) and cat (Section 12.2), and the loop will step through the words on standard input.

Here's an example:

for x in `cat`

do

    ...handle $x

done

Because this method splits the input into separate words, no matter how many words are on each input line, it can be more convenient than a while loop running the read command. When you use this script interactively, though, the loop won't start running until you've typed all of the input; using while read will run the loop after each line of input.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The normal Bourne shell for loop (Section 35.21) lets you take a list of items, store the items one by one in a shell variable, and loop through a set of commands once for each item:

for file in prog1 prog2 prog3

do

   ...process $file

done

I wanted a for loop that stores several different shell variables and makes one pass through the loop for each set of variables (instead of one pass for each item, as a regular for loop does). This loop does the job:

set Section 35.25

for bunch in "ellie file16" "donna file23" "steve file34"

do

   # PUT FIRST WORD (USER) IN $1, SECOND (FILE) IN $2...

   set $bunch

   mail $1 < $2

done

If you have any command-line arguments and still need them, store them in another variable before you use the set command. Or you can make the loop this way:

for bunch in "u=ellie f=file16 s='your files'" \

    "u=donna f=file23 s='a memo'" "u=steve f=file34 s=report"

do

   # SET $u (USER), $f (FILENAME), $s (SUBJECT):

   eval $bunch

   mail -s "$s" $u < $f

done

This script uses the shell's eval (Section 27.8) command to rescan the contents of the bunch variable and store it in separate variables. Notice the single quotes, as in

s='your

files'

; this groups the words for eval. The shell removes the single quotes before it stores the value into the s variable.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Almost every Unix command can use relative and absolute pathnames (Section 31.2) to find a file or directory. There are times you'll need part of a pathname — the head (everything before the last slash) or the tail (the name after the last slash). The utilities basename and dirname , available on most Unix systems, handle that.

The basename command strips any "path" name components from a filename, leaving you with a "pure" filename. For example:

% basename /usr/bin/gigiplot

gigiplot

% basename /home/mikel/bin/bvurns.sh

bvurns.sh

basename can also strip a suffix from a filename. For example:

% basename /home/mikel/bin/bvurns.sh .sh

bvurns

The dirname command strips the filename itself, giving you the "directory" part of the pathname:

% dirname /usr/bin/screenblank

/usr/bin

% dirname local

.

If you give dirname a "pure" filename (i.e., a filename with no path, as in the second example), it tells you that the directory is . (the current directory).

dirname and basename have a bug in some implementations. They don't recognize the second argument as a filename suffix to strip. Here's a good test:

% basename 0.foo .foo

If the result is 0, your basename implementation is good. If the answer is 0.foo, the implementation is bad. If basename doesn't work, dirname won't, either.

Here's an example of basename and dirname. There's a directory tree with some very large files — over 100,000 characters. You want to find those files, run split (Section 21.9) on them, and add huge. to the start of the original filename. By default, split names the file chunks xaa, xab, xac, and so on; you want to use the original filename and a dot (.) instead of x:

|| Section 35.14, exit Section 35.16

for path in `find /home/you -type f -size +100000c -print`

do

    cd `dirname $path` || exit

    filename=`basename $path`

    split $filename $filename.

    mv -i $filename huge.$filename

done

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

I used to think that the Bourne shell's while loop (Section 35.15) looked like this, with a single command controlling the loop:

while command

do

     ...whatever

done

But command can actually be a list of commands. The exit status of the last command controls the loop. This is handy for prompting users and reading answers. When the user types an empty answer, the read command returns "false" and the loop ends:

while echo -e "Enter command or CTRL-d to quit: \c"

      read command

do

      ...process $command

done

You may need a -e option to make echo treat escaped characters like \c the way you want. In this case, the character rings the terminal bell, however your terminal interprets that (often with a flash of the screen, for instance.)

Here's a loop that runs who and does a quick search on its output. If the grep returns nonzero status (because it doesn't find $who in $tempfile), the loop quits — otherwise, the loop does lots of processing:

while

    who > $tempfile

    grep "$who" $tempfile >/dev/null

do

    ...process $tempfile...

done

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This introduction is general and simplified. If you're a technical person who needs a complete and exact description, read a book on Unix programming.

Unix shells let you redirect the input and output of programs with operators such as > and |. How does that work? How can you use it better? Here's an overview.

When the Unix kernel starts any process (Section 24.3) — for example, grep, ls, or a shell — it sets up several places for that process to read from and write to, as shown in Figure 36-1.

Figure 36-1: Open standard I/O files with no command-line redirection

These places are called open files. The kernel gives each file a number called a file descriptor. But people usually use names for these places instead of the numbers:

The standard input or stdin (File Descriptor (F.D.) number 0) is the place where the process can read text. This might be text from other programs (through a pipe, on the command line) or from your keyboard.
The standard output or stdout (F.D. 1) is a place for the process to write its results.
The standard error or stderr (F.D. 2) is where the process can send error messages.

By default, as Figure 36-1 shows, the file that's opened for stdin, stdout, and stderr is /dev/tty — a name for your terminal. This makes life easier for users — and programmers, too. The user doesn't have to tell a program where to read or write because the default is your terminal. A programmer doesn't have to open files to read or write from (in many cases); the programs can just read from stdin, write to stdout, and send errors to stderr.

It gets better. When the shell starts a process (when you type a command at a prompt), you can tell the shell what file to "connect to" any of those file descriptors. For example, Figure 36-2 shows what happens when you run

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

By default, a command's standard error goes to your terminal. The standard output goes to the terminal or is redirected somewhere (to a file, down a pipe, into backquotes).

Sometimes you want the opposite. For instance, you may need to send a command's standard output to the screen and grab the error messages (standard error) with backquotes. Or you might want to send a command's standard output to a file and the standard error down a pipe to an error-processing command. Here's how to do that in the Bourne shell. (The C shell can't do this, although tcsh can.)

File descriptors 0, 1, and 2 are, respectively, the standard input, standard output, and standard error (Section 36.15 explains). Without redirection, they're all associated with the terminal file /dev/tty (Section 36.15). It's easy to redirect any descriptor to any file — if you know the filename. For instance, to redirect file descriptor 2 to errfile, type:

$ command

                2>errfile

You know that a pipe and backquotes also redirect the standard output:

$ command

                | ...

$ var=`

               command

               `

But there's no filename associated with the pipe or backquotes, so you can't use the 2> redirection. You need to rearrange the file descriptors without knowing the file (or whatever) that they're associated with. Here's how. You may find it useful to run this short Perl script, which simply prints "stdout" to standard output, and "stderr" to standard error:

#!/usr/bin/perl



print STDOUT "stdout\n";

print STDERR "stderr\n";

Let's start slowly. We will combine both standard output and standard error, sending them both as output, to be used as the input to a pipe or as the output of backquotes. The Bourne shell operator n >& m rearranges the files and file descriptors. It says, "Make file descriptor n point to the same file as file descriptor m." Let's use that operator on the previous example. We'll send standard error to the same place standard output is going:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Q: What is the difference between sh < file and sh file?

A:The first way keeps the script from reading anything else from its input. Consider the stdin-demo script:

while read word

do

   echo $word | sed s/foo/bar/

done

If run as sh stdin-demo, it will read from your terminal, replacing foo with bar. If run as

sh <

stdin-demo

, it will exit right away, since after reading the script, there's no input left.

— CT

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 36.17

The shell can read commands from its standard input or from a file. To run a series of commands that can change, you may want to use one program to create the command lines automatically — and pipe that program's output to a shell, which will run those "automatic" commands.

Here's an example. You want to copy files from a subdirectory and all its subdirectories into a single directory. The filenames in the destination directory can't conflict; no two files can have the same name. An easy way to name the copies is to replace each slash (/) in the file's relative pathname with a minus sign (-). For instance, the file named lib/glob/aprog.c would be copied to a file named lib-glob-aprog.c. You can use sed (Section 34.2) to convert the filenames and output cp commands like these:

cp from/lib/glob/aprog.c to/lib-glob-aprog.c

cp from/lib/glob/aprog.h to/lib-glob-aprog.h

    ...

However, an even better solution can be developed using nawk (Section 20.11). The following example uses find (Section 9.1) to make a list of pathnames, one per line, in and below the copyfrom directory. Next it runs nawk to create the destination file pathnames (like to /lib-glob-aprog.c) and write the completed command lines to the standard output. The shell reads the command lines from its standard input, through the pipe.

This example is in a script file because it's a little long to type at a prompt. But you can type commands like these at a prompt, too, if you want to:

#!/bin/sh

find copyfrom -type f -print |

awk '{

    out = $0

    gsub("/", "-", out)

    sub("^copyfrom-", "copyto/", out)

    print "cp", $0, out

}' |

sh

If you change the last line to sh -v, the shell's verbose option (Section 37.1) will show each command line before executing it. If the last line has sh -e , the shell will quit immediately after any command returns a nonzero exit status

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you need to quote your hereis document (Section 27.16) terminators, there's an annoying problem: sh and csh demand different conventions. If you are using sh, you must not quote the terminator. For example,

#! /bin/sh

cat << 'eof'

Hi there.

eof

If you are using csh, however, you must quote the terminator. The following script prints three lines, not one:

#! /bin/csh

cat << \eof

Hi.  You might expect this to be the only line, but it's not.

eof

'e'of

\eof

— CT

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you type your password, Unix turns off echoing so what you type won't show on the screen. You can do the same thing in shell scripts with

stty

-echo

.

stty Section 5.7, read Section 35.18

#!/bin/sh

   ...

trap 'stty echo; exit' 0 1 2 3 15

# use the right echo for your Unix:

echo "Enter code name: \c"

#echo -n "Enter code name: "

stty -echo

read ans

stty echo

   ...

The response is stored in $ans. The trap (Section 35.17) helps to make sure that, if the user presses CTRL-c to abort the script, characters will be echoed again.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

expr is a very handy tool in shell programming, since it provides the ability to evaluate a wide range of arithmetic, logical, and relational expressions. It evaluates its arguments as expressions and prints the result.

Here's the syntax. The [brackets] mean "optional"; don't type the brackets:

expr arg1 operator arg2 [ operator arg3 ... ]

Arguments and operators must be separated by spaces. In many cases, an argument is an integer, typed literally or represented by a shell variable. There are three types of operators: arithmetic, relational, and logical.

Exit status (Section 35.12) values for expr are 0 if the expression evaluates nonzero and non-null, 1 if the expression evaluates to 0 or null, and 2 if the expression is invalid.

Arithmetic operators

Use these to produce mathematical expressions whose results are printed:

+: Add arg2 to arg1.
-: Subtract arg2 from arg1.
*: Multiply the arguments.
/: Divide arg1 by arg2.
%: Take the remainder when arg1 is divided by arg2 (modulus).

Addition and subtraction are evaluated last, unless they are grouped inside parentheses. The symbols *, (, and ) have meaning to the shell, so they must be escaped (preceded by a backslash or enclosed in quotes).

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The expr (Section 36.21) command does a lot of different things with expressions. One expression it handles has three arguments: first, a string; second, a colon (:); third, a regular expression (Section 32.4). The string and regular expression usually need quotes.

expr can count the number of characters that match the regular expression. The regular expression is automatically anchored to the start of the string you're matching, as if you'd typed a ^ at the start of it in grep, sed, and so on. expr is usually run with backquotes (Section 28.14) to save its output:

$ part="resistor 321-1234-00"

$ name="Ellen Smith"

   ...

$ expr "$part" : '[a-z ]*[0-9]'        

               ...character position of first number

10

$ len=`expr "$name" : '[a-zA-Z]*'`

$ echo first name has $len characters

first name has 5 characters

When a regular expression matches some character(s), expr returns a zero ("true") exit status (Section 35.12). If you want a true/false test like this, throw away the number that expr prints and test its exit status:

/dev/null Section 43.12

$ if expr "$part" : '.*[0-9]' > /dev/null

> then echo \$part has a number in it.

> else echo "it doesn't"

> fi

$part has a number in it.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

How can you parse (split, search) a string of text to find the last word, the second column, and so on? There are a lot of different ways. Pick the one that works best for you — or invent another one! (Unix has lots of ways to work with strings of text.)

The expr command (Section 36.21) can grab part of a string with a regular expression. The example below is from a shell script whose last command-line argument is a filename. The two commands below use expr to grab the last argument and all arguments except the last one. The "$*" gives expr a list of all command-line arguments in a single word. (Using "$@" (Section 35.20) here wouldn't work because it gives individually quoted arguments. expr needs all arguments in one word.)

last=`expr "$*" : '.* \(.*\)'`     # LAST ARGUMENT

first=`expr "$*" : '\(.*\) .*'`    # ALL BUT LAST ARGUMENT

Let's look at the regular expression that gets the last word. The leading part of the expression, .* , matches as many characters as it can, followed by a space. This includes all words up to and including the last space. After that, the end of the expression, $.*$, matches the last word.

The regular expression that grabs the first words is the same as the previous one — but I've moved the

\(

\)

pair. Now it grabs all words up to but not including the last space. The end of the regular expression, .*, matches the last space and last word — and expr ignores them. So the final .* really isn't needed here (though the space is). I've included the final .* because it follows from the first example.

expr is great when you want to split a string into just two parts. The .* also makes expr good for skipping a variable number of words when you don't know how many words a string will have. But expr is poor at getting, say, the fourth word in a string. And it's almost useless for handling more than one line of text at a time.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 28.14 introduced command substitution with a pair of backquotes (''). Let's review. The shell runs a backquoted string as a command, then replaces the string with its output. Sometimes — though not as often — you'll want to use the results from one backquoted string as arguments to another command, itself also inside backquotes. To do that, you need to nest the backquotes to tell the shell which command (which set of backquotes) should be done first, with its output given to the second command. This is tricky with backquotes; the Korn shell introduced an easier way that you'll see below. Here's a simple example — the first command line uses nested backquotes, and the next two commands show its parts:

$ echo "Next year will be 200`expr \`date +%y\` + 1`."

Next year will be 2002.

$ date +%y

01

$ expr 01 + 1

2

The command to run first has escaped backquotes (\'\') around it. In the example above, that's the

date

+%y

command. date +%y outputs the year — in this case, 01 — and that value is passed to the expr command. expr adds 01 and 1 to get 2. Then that result (from the outer backquotes) is passed to echo, on its command line, and echo prints the message.

Why does the inner command, inside the escaped backquotes (\'\'), run first? It's because the backslash before the backquote turns off the special meaning (Section 27.12) of the backquote. When the shell first evaluates the command line, which backquotes does it see? It sees the unescaped backquotes, the ones around the expr command, and the shell runs the command:

expr `date +%y` + 1

But when the shell evaluates that command line, it sees the backquotes in it (now unescaped) and runs that command — date +%y. The

date

+%y

command outputs 01. Next, the shell can finish the command expr 01 + 1. It outputs 2. Then the echo command can print its message.

Whew. Most newer Bourne-type shells have an easier way: the $( command ) operators. Use $( before the command, where you would use an opening backquote. Put the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The shell's case statement (Section 35.10) has some advantages over the test command (Section 35.26) — for instance, case can do pattern matching. But test has the -a and -o "and" and "or" operators; those don't seem easy to do with case. And test isn't built in to some older shells, so using case may be faster.

Here's a way to test two things with one case statement. It won't solve all your problems. If you think carefully about the possible values the variables you're testing can have, though, this might do the trick. Use a separator (delimiter) character between the two variables.

In the example below, I've picked a slash (/). You could use almost any character that isn't used in case pattern matching (Section 35.11) and that won't be stored in either $# or $1. The case below tests the command-line arguments of a script:

case "$#/$1" in

1/-f) redodb=yes ;;

0/) ;;

*)  echo "Usage: $0 [-f]" 1>&2; exit 1 ;;

esac

If there's one argument ($# is 1) and the argument ($1) is exactly -f, the first pattern matches, and the redodb variable is set. If there's no argument, $# will be 0 and $1 will be empty, so the second pattern matches. Otherwise, something is wrong; the third pattern matches, the script prints an error and exits.

Of course, you can do a lot more this way than just testing command-line arguments.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix has a lot of ways to output text from the command line into the terminal (or window) where a script is running. But there are times you'd like to pop open a new window (under the X Window System (Section 1.22)), give the user a message — and maybe let the user reply too. X comes with a standard client named xmessage that does this. It pops open a window like Figure 36-7 with a message, then waits for the user to click a button (possibly one of many) or press RETURN. For details, you can read the xmessage manual page. I'll show how I integrated xmessage into a shell script.

Figure 36-7: An xmessage window from xwrist

Go to http://examples.oreilly.com/upt3 for more information on: xwrist

A good way to prevent wrist injuries (from too much typing) is by taking periodic breaks. The xwrist script uses xmessage to remind me (every 10 minutes) to take a break — and prints a fortune for me to read while I do.

Let's look at two parts of the script. First, the script checks to see if the X Window System DISPLAY environment variable (Section 35.5) is set; if not, it complains (with a message like

xwrist: DISPLAY: unset? I

only work under the X Window System

) and exits:

: Section 36.6, ${..?..} Section 36.7

: ${DISPLAY?"unset? I only work under the X Window System"}

After checking the command-line arguments and setting some shell variables, the script does its main work with an endless loop:

`...`Section 28.14

while sleep $delay

do

    if xmessage -nearmouse -geometry $geometry -title "$title" \

        -buttons okay:1,quit:0 -default okay \

        "`/usr/games/fortune | fmt $fmtarg`"

    then exit 0

    fi

done

The while loop ( Section 35.15) is endless because sleep normally returns 0 (Section 35.12). As long as the user keeps clicking the okay button, a new xmessage window will pop up again $delay seconds after the previous one. The

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's an efficient and portable way to create a lockfile from a shell script. It's also an interesting demonstration of the way that Unix umasks and file permissions (Section 50.2) are handled.

A lockfile can be used when a particular program might be run more than once at the same time and you need to be sure that only one instance of the program can do something (modify some file, access a printer, etc.). To really do this right, the program needs to both test for the lockfile and create it (if it doesn't exist) in one atomic operation. If the test-and-set operation isn't atomic — for instance, if a program tests for the lock file in one command and then creates the lock file in the next command — there's a chance that another user's program could do its test at the precise moment between the first program's (non-atomic) test and set operations. The technique in this article lets you make a lockfile atomically from a shell script.

This technique doesn't work for scripts run as the superuser (root). It depends on the fact that a standard user can't write a file without write permisson. But root can write any file, whether it has write permission or not. If there's a chance that root might run your script, you might want to add a test of the UID — by running the id command, for instance — and be sure that the UID isn't 0 (the superuser's).

Let's say you have a script called edmaster; it edits a master configuration file named config. To be sure that two users can't modify the config file at the same time, the first edmaster checks whether the lockfile exists. If the lockfile doesn't exist, edmaster creates it and modifies the config file. When it's done editing, it removes the lockfile. If someone tries to run a second edmaster process, it sees the lockfile from the first edmaster, waits, and checks every few seconds to see if the lockfile is gone. Once the first edmaster removes the lockfile, the second edmaster can create the lockfile and do its editing of config. (Note that some editors — for instance,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Depending on the Bourne shell version you have, the error messages it gives can be downright useless. For instance, it might say just End of file unexpected. Here are a few tricks to use to get a little more information about what's going on. Remember, it's probably best for you to use one of shells derived from the Bourne shell, rather than the C shell, for scripting.

Start your script like this:

#!/bin/sh -xv

(If your Unix can't handle #!, use the command set -xv (Section 35.25)). The -xv shows you what's happening as the shell reads your script. The lines of the script will be shown as the shell reads them. The shell shows each command it executes with a plus sign (+) before the command.

Note that the shell reads an entire loop (for, while, etc.) before it executes any commands in the loop.

If you want to run a script with debugging but you don't want to edit the script file, you can also start the shell explicitly from the command line and give the options there:

% sh -xv 

                  scrfile

Debugging output is usually pretty long, more than a screenful, so I pipe it to a pager like less. But the shell sends its debugging output to stderr, so I pipe both stdout and stderr (Section 43.4) to the pager.

$ scrfile

                   2>&1 | less

Do you want to save the debugging output in a file and see it on your screen, too? Use tee (Section 43.8) to copy the scrfile stdout and stderr; add tee to the pipeline before the pager.

$ scrfile

                   | tee 

                  outfile 

                  2>&1 | 

                  less

If the script is slow, you can run it in the background. Redirect the shell's output and errors (Section 43.5, Section 27.11) into a temporary file. Use tail -f (Section 12.10) to "watch" the log file. If you want to do something else while the script runs, just kill the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Depending on the Bourne shell version you have, the error messages it gives can be downright useless. For instance, it might say just End of file unexpected. Here are a few tricks to use to get a little more information about what's going on. Remember, it's probably best for you to use one of shells derived from the Bourne shell, rather than the C shell, for scripting.

Start your script like this:

#!/bin/sh -xv

(If your Unix can't handle #!, use the command set -xv (Section 35.25)). The -xv shows you what's happening as the shell reads your script. The lines of the script will be shown as the shell reads them. The shell shows each command it executes with a plus sign (+) before the command.

Note that the shell reads an entire loop (for, while, etc.) before it executes any commands in the loop.

If you want to run a script with debugging but you don't want to edit the script file, you can also start the shell explicitly from the command line and give the options there:

% sh -xv 

                  scrfile

Debugging output is usually pretty long, more than a screenful, so I pipe it to a pager like less. But the shell sends its debugging output to stderr, so I pipe both stdout and stderr (Section 43.4) to the pager.

$ scrfile

                   2>&1 | less

Do you want to save the debugging output in a file and see it on your screen, too? Use tee (Section 43.8) to copy the scrfile stdout and stderr; add tee to the pipeline before the pager.

$ scrfile

                   | tee 

                  outfile 

                  2>&1 | 

                  less

If the script is slow, you can run it in the background. Redirect the shell's output and errors (Section 43.5, Section 27.11) into a temporary file. Use tail -f (Section 12.10) to "watch" the log file. If you want to do something else while the script runs, just kill the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you have a shell script that sets several variables and you want to show the value of one of them, you can add a loop that asks you for variable names and displays their values (Section 36.14):

% cat myscript

#!/bin/sh

   ...

while echo "Pick a variable; just RETURN quits: \c"

      read var

do

      case "$var" in

      "") break ;;

      *)  eval echo \$$var ;;

      esac

done

The loop prompts Pick a variable:, then reads a value; if you type an empty answer, the loop quits. Otherwise, the value of that variable is displayed; the eval (Section 27.8) command scans the echo command line twice.

This tip isn't just good for debugging. It's good in any shell script where you need to show the value of a variable by typing its name.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The test and [ (square bracket) commands ( Section 35.26) can compare two numbers. But it's an error if one of the numbers you test is stored in a shell variable that's empty or doesn't exist. For example, an empty num variable here will give you a Syntax error:

if [ "$num" -gt 0 ]

then ...

To stop syntax errors, add a leading zero, like this:

if [ "0$num" -gt 0 ]

then ...

In that case, if $num is empty, the test will compare 0 to 0. If $num is 1, the test will be true (because 01 is greater than 0) — and so on, just as it should be.

The zero trick doesn't work with negative numbers, though, so if you expect ever to need to deal with negative numbers, you may want to look into other methods of checking to see if a variable has a value, such as this method from the bash shell, which displays an error if the variable is null or unset, or the following method, which assigns a default value:

#!/bin/sh

   ...



# check $num first, fail with error

tmp=${num:?"num not set"}



# use a default

default=0

if [ ${num:-default} -gt 0 ]

then

   ...

—JP and SJC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Using the test or [ (square bracket) command (Section 35.26) for a string test can cause errors if the variable starts with a dash (-). For example:

if [ "$var" = something ]

then ...

If $var starts with -r, the test command may think that you want to test for a readable file.

One common fix (that doesn't always work; see below) is to put an extra character at the start of each side of the test. This means the first argument will never start with a dash; it won't look like an option:

if [ "X$var" = Xsomething ]

then ...

That trick doesn't work if you want the test to fail when the variable is empty or not set. Here's a Bourne shell test that handles empty variables:

case "${var+X}" in

X) ...do this if variable is set...

   ;;



*) ...do this if variable is not set...

   ;;

esac

If $var is set (even if it has an empty string), the shell replaces ${var+X} (Section 36.7) with just X and the first part of the case succeeds. Otherwise the default case, *), is used.

See also Section 37.3 for a brief example of bash parameter expansion and dealing with unset or null values by reporting an error or by assigning default values.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Q: I need to pass a shell script some arguments with multiple words. I thought that putting quotes (Section 27.12) around command-line arguments would group them. The shell script seems to ignore the quoting, somehow. Here's a simple example:

$ cat script

   ...

for arg in $*

do

    echo "Argument is $arg"

done

$ script '1 2 3' 4

   ...

Argument is 1

Argument is 2

Argument is 3

Argument is 4

A: This is the way $* is defined to work. $* expands to:

$1 $2

[not "$1" "$2;" — JP] if there are two arguments. Hence the for loop reads:

for arg in 1 2 3 4

Note that the quotes are gone. What you wanted the shell to see was:

for arg in '1 2 3' 4

You can't get that, but you can get something that is good enough:

"$@" Section 35.20

for arg in "$@"

In effect, $@ expands to:

$1" "$2

Putting ""s around $@, the effect is:

for arg in "$1" "$2"

Shell quoting is unnecessarily complex. The C shell actually has the right idea (variables can be set to "word lists"; argv is such a list), but its defaults and syntax for suppressing them make for an artless programming language:

foreach arg ($argv:q)      # colon q ?!?

For the special case of iterating a shell variable over the argument list as it stands at the beginning of the iteration, the Bourne shell provides the construct for arg do [i.e., no in list — JP]:

for arg

do echo "Argument is $arg"

done

The example produces:

Argument is 1 2 3

Argument is 4

"$@" is still needed for passing argument lists to other programs. Unfortunately, since $@ is defined as expanding to:

$1" "$2...$n-1" "$n

(where n is the number of arguments), when there are no arguments, "$@" expands to "", and "" produces a single argument. [Many Unix vendors considered this a bug and changed it so that it produces

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Like all other operating systems, Unix has a concept of the time. And virtually all Unix systems, even the smallest, include a clock with some sort of battery backup built in.

All Unix systems keep time by counting the number of microseconds since midnight, January 1, 1970, Greenwich Mean Time. This date is commonly called the epoch , and it has folk-significance as the begining of the Unix era. Although the first work on Unix began in the late '60s, the first versions of Unix were available (within Bell Laboratories) in the early '70s.

This count gets updated roughly 60 times per second. The exact rate depends on your particular Unix system and is determined by the constant, HZ, defined in the header file /usr/include/sys/param.h:

#define   HZ   60

This is the time's "resolution," often referred to as the clock's "tick." Note that it has nothing to do with your system's CPU clock rate. Time measurements are normally no more precise than your system's clock resolution, although some systems have added facilities for more precise timing.

If your Unix system belongs to a network, it is important to keep all the clocks on the network "in sync." Strange things happen if you copy a file from one system to another and its date appears to be some time in the future. Many Unix systems run a time daemon (one of those mysterious helper programs (Section 1.10)) to take care of this.

Unix automatically keeps track of daylight savings time (summer time), leap years, and other chronological trivia. When the system is installed, you have to tell it your time zone and the style of daylight savings time you want to observe. As Unix has become an international standard, the number of time zones (and obscure ways of handling daylight savings time) it can handle correctly has proliferated. In a few cases, you still have to handle these things by hand; for example, in Europe, as of this writing, the beginning and end of Summer Time were set periodically by the European Parliament, and so may change. Care for Libyan Standard Time?

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Are you typing a complicated set of commands that you need to show someone else or keep "on file" for documentation? Are you debugging a program that goes wrong somewhere — but the error message flashes by so fast that you can't see it? Do you want to show a "prerecorded" demonstration of an interactive program? The script program can help with all of these problems.

Versions of script on Unix systems without ptys aren't as flexible as the version I'm explaining here. For instance, those versions won't let you use job control (Section 23.3) during the script.

To copy everything you do into a file, just type:

% script

Script started, file is typescript

%

Now you can type any Unix command that you'd use at a shell prompt. Everything you do is copied into a file named typescript in the current directory. (To use a different filename, type its pathname (Section 1.16) on the command line, like script scriptfile.) When you're done, type CTRL-d or exit (Section 24.4) at a shell prompt.

One thing that surprises people is that everything will be copied into the script file. That includes escape sequences that programs send to your terminal. This is both good and bad.

The good part is that you can "play back" whatever happened by catting (Section 12.2) the script to your screen. When things get boring, you can run an interactive program like vi inside the script — then quit the script and play it back with cat typescript. The cursor will fly across the screen and your file will be re-edited before your eyes. (This is easier to see if the terminal is set to a slow data rate.)

The bad part is that errors you correct and other terminal-control sequences will be in the file, too. If you edit or print the script file, it may be full of "junk" such as ^M (carriage return) and ^H (backspace) characters. (A command like cat -v or od -c (Section 12.4) will show you these characters.) If the file has just a few of these characters, you can clean it up by hand with your text editor's global substitution commands. You can also automate your "script cleaning" with techniques such as the ones in Section 21.11 and Section 37.8.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As Section 37.7 explains, the files made by the script program can have stray control characters in them. The shell script called script.tidy can clean them up. Dan Bernstein wrote it and posted it to Usenet; I made a few changes. It reads from files or standard input and writes to standard output.

Go to http://examples.oreilly.com/upt3 for more information on: script.tidy

script.tidy uses the sed (Section 34.1) substitute command to remove CTRL-m (RETURN) characters from the ends of lines. It uses the sed test command (Section 34.21) to repeat a series of commands that delete a character followed by CTRL-h (BACKSPACE). If you use DELETE as your erase character (Section 5.8), change the script to eat DELETE instead of BACKSPACE. script.tidy uses a trick with echo and tr to store the control characters in shell variables. Because the sed script has doublequotes (Section 27.12) around it, the shell variables are substituted in the right places before the shell starts sed.

eval Section 27.8, exec Section 36.5

#!/bin/sh



# Public domain.



# Put CTRL-M in $m and CTRL-H in $b.

# Change \010 to \177 if you use DEL for erasing.

eval `echo m=M b=H | tr 'MH' '\015\010'`

exec sed "s/$m\$//

:x

s/[^$b]$b//

t x" $*

You can also hack the sed script in script.tidy to delete some of your terminal's escape sequences. (A really automated script.tidy would read your termcap or terminfo entry and look for all those escape sequences in the script file.)

Bear in mind that script was designed to emulate a paper terminal; if you've modified your prompt, especially if you are using multiple-line prompts, your script output is going to be full of far worse junk than script.tidy can fix. If you find that script simply doesn't do it for you, you should consider whether you want a complete record of all terminal input and output or just a record of what you typed. If the latter is more to your liking, you should look into the various history editing and printing capabilities provided by modern shells.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The yes command ( Section 14.5) outputs text over and over. If you need a file of some size for testing, make it with yes and head ( Section 12.12). For example, to make a file 100k (102,400) characters long, with 12,800 8-character lines (7 digits and a newline), type:

% yes 1234567 | head -12800 > 100k-file

On some Unix systems, the command may "hang" and need to be killed with CTRL-c because head keeps reading input from the pipe. If it hangs on your system, replace

head

-12800

with sed 12800q.

You might just want to use perl, instead:

$ perl -e 'print "1234567\n" x 12800' > file

For the Unix admin who has everything, here's one more way, this time using the venerated dd command:

$ yes | dd of=file count=25

There are many variations on this theme. The preceding example simply copies 25 blocks of 512 bytes each from standard input (the output of the yes command) to the file file. You could also specify a number of bytes to read at a time, using the ibs option, and then specify the number of records to write out, using count :

$ yes | dd ibs=1 of=file count=12800

There's More Than One Way To Do It. Be careful, though — you can fill up a disk pretty quickly playing around with the dd command!

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Making copies of critical files in case the originals become inaccessible is called backing them up or making backups. Backups are insurance. They are time and effort you spend protecting yourself from things that might never happen. Your hard drive might never crash, but what vital things would you lose if it did?

Exactly what "making a backup" means varies depending on your circumstances. All of the following examples are ways to make backups applicable to some specific environment:

Copying some files onto another disk on the same machine, so that if one hard drive dies you still have a copy. (A more sophisticated and automatic way of doing this, which you may have heard about, is called Redundant Array of Inexpensive Disks or RAID.)
Making a compressed tar file and copying it to another machine, so that if one machine crashes you still have a copy.
Writing copies of your files to a Zip drive, CD-RW, or DVD-RW.
tarring (Section 38.2) files to a tape.
Nightly automatic backups of everything that's changed that day (called an incremental backup) to multiple tapes, with copies of the tapes stored in offsite secure storage.

If you are just trying to protect your files on your personal machine, simply making sure that critical files have copies on multiple physical disks or occasionally copying files onto another machine or removable storage is probably sufficient. If you're administering a machine that has multiple users, regular backups are almost certainly a necessity. If those users are doing business-critical tasks, very regular backups and off-site copies are a requirement to protect the investment of time involved.

— DJPH

When many Unix users think of file archives, on tape or in an archive file, they think of the tar utility. There are other ways to make archives and handle tapes — including dump and dd. This article summarizes articles about tar in this chapter and others.

Although tar is a tape archiver, one of its common uses is making an archive file on disk

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Making copies of critical files in case the originals become inaccessible is called backing them up or making backups. Backups are insurance. They are time and effort you spend protecting yourself from things that might never happen. Your hard drive might never crash, but what vital things would you lose if it did?

Exactly what "making a backup" means varies depending on your circumstances. All of the following examples are ways to make backups applicable to some specific environment:

Copying some files onto another disk on the same machine, so that if one hard drive dies you still have a copy. (A more sophisticated and automatic way of doing this, which you may have heard about, is called Redundant Array of Inexpensive Disks or RAID.)
Making a compressed tar file and copying it to another machine, so that if one machine crashes you still have a copy.
Writing copies of your files to a Zip drive, CD-RW, or DVD-RW.
tarring (Section 38.2) files to a tape.
Nightly automatic backups of everything that's changed that day (called an incremental backup) to multiple tapes, with copies of the tapes stored in offsite secure storage.

If you are just trying to protect your files on your personal machine, simply making sure that critical files have copies on multiple physical disks or occasionally copying files onto another machine or removable storage is probably sufficient. If you're administering a machine that has multiple users, regular backups are almost certainly a necessity. If those users are doing business-critical tasks, very regular backups and off-site copies are a requirement to protect the investment of time involved.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When many Unix users think of file archives, on tape or in an archive file, they think of the tar utility. There are other ways to make archives and handle tapes — including dump and dd. This article summarizes articles about tar in this chapter and others.

Although tar is a tape archiver, one of its common uses is making an archive file on disk (Section 39.2). Because tar "pads" its archives with NUL characters, on-disk tar archive files can be much bigger than the size of the individual files put together. Both to fix that and generally to save space, tar files are often compressed. The GNU tar ( Section 39.3) can compress files while storing them and uncompress them while reading them, automatically. If you don't have GNU tar, you may need to uncompress an archive manually. Note that a compressed tar archive can take less disk space ( Section 15.7) than compressing individual small files.
Because tar keeps most of a file's inode information, it can make a more complete copy (Section 10.13) of a file or directory tree than utilities such as cp.
Yes, we do have articles about archives on tape. Section 38.3 has enough information to make your own archive, although you might need the details from Section 38.5, too. After you've made an archive, you'll probably want to restore it, at least as a test to be sure your archive is okay. Section 38.6 explains how.
If there isn't a tape drive on your computer, read Section 38.7 about using a drive on another computer.
tar copies a directory tree, recursively, from top to bottom. What if you don't want to archive everything? You can back up just some files by combining ls -lt and find. Some versions of tar have options for including or excluding certain files and directories (Section 39.3).

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As someone who has been an end user and a system administrator, I strongly believe that every user should understand the importance of backups.

If you have data that is important to you, you should have a known backup.

Accidents and oversights happen. Tapes can be damaged, lost, or mislabeled. Assume that your system administrator is top-notch. The best administrator can recover your lost data 99 percent of the time. There is still a small chance that the files you need might not be recovered. Can you afford to duplicate months of effort 1 percent of the time? No.

An experienced user learns to be pessimistic. Typically, this important perspective is learned the hard way. Perhaps a few hours are lost. Perhaps days. Sometimes months are lost.

Here are some common situations:

A user works on a file all day. At the end of the day, the file is deleted by accident. The system manager cannot recover the file. A day's work has been lost.
A programmer tries to clean up a project directory. Instead of typing rm *.o the programmer types rm * .o and the entire directory is lost.
A user deletes a file by accident. After a few days, the user asks the system administrator to recover the file. The incremental backup system has reused the only tape the missing file was on.
A large project is archived on a magnetic tape and deleted from the disk. A year later, some of the information is needed. The tape has a bad block at the beginning. The system manager must learn how to recover data from a bad tape. The attempt is often unsuccessful. The information is lost forever, and must be re-created at the cost of months of effort.
Someone breaks into a computer and alters or deletes crucial information.
A fire breaks out in the computer room. The disks and all of the backup tapes are lost.

Gulp! I scared myself. Excuse me for a few minutes while I load a tape...

Ah! I feel better now. As I was saying, being pessimistic has its advantages.

Making a backup is easy. Get a blank tape and put a label on it. Learn how to load it into the tape drive. Then do the following:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 38.3 explains the minimal basics of using tar to make backups, but there are lots of variations that can be very useful.

To create a tar archive for copying to another disk or another machine:

% tar cvf 20020214-book.tar ./book

tar 's c option stands for create, v for verbose, and the f option for file. 20020214-book.tar is the new archive file to create, and ./book says to archive the directory book in the current directory. Once you have an archive, you might want to compress it to save space. gzip and bzip2 are your best bets. (I use bzip2 here largely because it tends to give better compression, but be aware that gzip is more widely available and thus may be safer for backups.) You can compress it once you've made it:

% ls -l 20020214-book.tar

-rw-r--r--  1 deb  deb  19415040 Feb 14 23:15 20020214-book.tar

% bzip2 20020214-book.tar

% ls -l 20020214-book.tar.bz2

-rw-r--r--  1 deb  deb   4033775 Feb 14 23:15 20020214-book.tar.bz2

Or you can compress it as you make it. GNU tar supports gzip compression on the fly with the z or - -gzip options and bzip2 compression on the fly with the - -bzip2 option, or you can pipe into gzip or bzip2:

% tar czvf 20020214-book.tar.gz ./book



% tar cvf 20020214-book.tar.bz2 --bzip2 ./book



% tar cvf - ./book | bzip2 > 20020214-book.tar.bz2

Section 39.2 and Section 39.3 have more information on using tar.

You can get more protection from certain kinds of mishaps by using a version control system like RCS (Section 39.5) or CVS (Section 39.7) to save every version of a file you are updating frequently. While it doesn't protect you from disk crashes, a version control system provides the ability to back up to a previous version if something gets changed or deleted incorrectly.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This article was written for Linux systems, but the advice applies everywhere. You may need to make some adjustments — in the names of the tape drive devices and some filesystem directories, for instance. If you're making personal backups (of the files on your account, for instance), you can substitute your directory names for the system directories covered here, but the command names and techniques won't change.

As Section 38.3 says, the simplest way to make a backup is to use tar to archive all the files on the system or only those files in a set of specific directories. Before you do this, however, you need to decide what files to back up. Do you need to back up every file on the system? This is rarely necessary, especially if you have your original installation disks or CD-ROM. If you have made specific, important changes to the system, but everything else could simply be reinstalled in case of a problem, you could get by archiving only those files you have made changes to. Over time, however, it is difficult to keep track of such changes.

In general, you will be making changes to the system configuration files in /etc. There are other configuration files as well, and it can't hurt to archive directories such as /usr/local (where various packages generally get installed) and /usr/X11R6/lib/X11 (which contains the X Window System configuration files). You may want to do filtering on these directories and back up only the configuration files, since binaries in /usr/local and things like fonts in the X11 distribution can be reinstalled from their original packages easily enough.

You should also back up your kernel sources (if you have patched your kernel sources); these are found in /usr/src/linux (/usr/src/sys on *BSD). At the very least, you'll want to back up your kernel configuration file if you've built your own kernel; it's in /usr/src/linux/.config (or /usr/src/sys/ platform/conf/KERNELNAME on *BSD).

It's a good idea to keep notes on what features of the system you've changed so you can make intelligent choices when making backups. If you're truly paranoid, go ahead and back up the whole system: that can't hurt, but the cost of backup media might.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you create an archive, there are several ways to specify the directory. If the directory is under the current directory, you could type:

% tar c project

A similar way to specify the same directory is:

% tar c ./project

If you are currently in the directory you want archived, you can type:

% tar c .

Another way to archive the current directory is to type:

% tar c *

Here, the shell expands the asterisk (*) to the files in the current directory. However, it does not match files starting with a dot (.), which is why the previous technique is preferred.

This causes a problem when restoring a directory from a tar archive. You may not know whether an archive was created using . or the directory name.

I always check the names of the files before restoring an archive:

% tar t

If the archive loads the files into the current directory, I create a new directory, change to it, and extract the files.

If the archive restores the directory by name, then I restore the files into the current directory.

If you want to restore a single file, get the pathname of the file as tar knows it, using the t flag. You must specify the exact filename, because filename and ./filename are not the same to tar. You can combine these two steps into one command by using:

% tar xvf /dev/rst0 `tar tf /dev/rst0 | grep 

                  filename`

Note that this may run very slowly, though, as the entire tar file has to be read once (and the tape rewound) before any restoration can happen. Be careful: you may also get a lot more than you expected; for example, if you're looking for README using this technique, you'd also get README.Solaris and everything in the doc/READMEs directory, possibly overwriting files you wanted to keep.

Whenever you use tar to restore a directory, you must always specify some filename. If none is specified, no files are restored.

There is still the problem of restoring a directory whose pathname starts with a slash (

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If your computer doesn't have a tape drive connected, creating tar (Section 38.2) backup files is slightly more complicated. If you have an account on a machine with a tape drive, and the directory is mounted via NFS ( Section 1.21), you can just rlogin (Section 1.21) to the other machine and use tar to back up your directory.

If the directory is not NFS mounted, or it is mounted but you have permission problems accessing your own files, you can use tar, rsh (Section 1.21), and dd (Section 21.6) to solve this dilemma. The syntax is confusing, but if you forget, you can use man tar (Section 2.1) to refresh your memory. The command to dump the current directory to a tape in a remote machine called zephyrus is:

% tar cvfb - 20 . | rsh zephyrus dd of=/dev/rmt0 obs=20b

Here, the output file of tar is -, which tar interprets as standard input if tar is reading an archive or standard output if tar is creating an archive.

The dd command copies data from standard input to the device /dev/rmt0.

This example assumes you can use rsh without requiring a password. You can add your current machine's name to the remote .rhosts file (Section 1.21) if you get a Password: prompt when you use rlogin to access this machine. You also can use ssh , which is generally more secure than rsh, and the ssh-agent utility to allow logins without a password.

— BB

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you're using GNU tar, you can probably ignore the tips in Section 38.7 about using a tape drive on a remote system. GNU tar makes it easy to access a remote drive via rsh or a similar command like ssh.

When referring to a local host, the GNU tar f option takes a plain filename like foo.tar or a device name like /dev/rmt0. If you put a colon (:) before that name, though, you can prepend a remote hostname — and, optionally, a username. For example, to get a table of contents of the tape on the drive /dev/rmt8 on the remote host server2, logging into server2 as yourself, type:

% tar tf server2:/dev/rmt8

To specify a different username than the one on your local host, add it with an @ before the hostname. (This assumes you're allowed to connect to the remote host without a password — because there's a .rhosts file on the remote system, for instance.) For example, to connect to server2 as heather and extract the files reports/products.sgml and reports/services.sgml from /dev/rmt8:

{ } Section 28.4

% tar xf

               heather@server2:/dev/rmt8 reports/{products,services}.sgml

By default, GNU tar uses rsh, remsh, or nsh to access the remote machine, though that can be changed when tar is built and installed on your host. If you want another access command, like ssh, you can set that with the - -rsh-command option. The next example gets the contents of the archive on the drive /dev/rmt8 from the host capannole.it using ssh. Note that tar doesn't check your search path (Section 27.6) for the rsh-command; you have to give its absolute pathname (which you can get with a command like which (Section 2.6)):

% tar -x --file=capannole.it:/dev/rmt8

               --rsh-command=/usr/bin/ssh

On the other hand, if you need to use a local filename with a colon in it, add the - -force-local option.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As I was working on this book, I was constantly editing lots of random files all through a directory tree. I archived some of the files in a revision control system (Section 39.4), but those archives, as well as the nonarchived files, still would be vulnerable if my disk crashed. (And naturally, close to a deadline, one hard disk started making whining noises...)

The answer I came up with was easy to use and simple to set up. It's a script named ptbk , and this article explains it. To run the script, I just type its name. It searches my directory tree for files that have been modified since the last time I ran ptbk. Those files are copied into a dated compressed tar archive and copied to a remote system using scp. The process looks like this:

$ ptbk

upt/upt3_changes.html

upt/BOOKFILES

upt/art/0548.sgm

upt/art/1420.sgm

upt/art/1430.sgm

upt/art/0524.sgm

upt/BOOKIDS

upt/ulpt3_table

Now copying this file to bserver:

-rw-rw-r--    1 jpeek    323740 Jan  3 23:08 /tmp/upt-200101032308.tgz

upt-200101032308.tgz     |     316 KB |  63.2 kB/s | ETA: 00:00:00 | 100%

The script actually doesn't copy all of the files in my directory tree. I've set up a tar exclude file that makes the script skip some files that don't need backing up. For instance, it skips any filename that starts with a comma (,). Here's the file, named ptbk.exclude:

upt/ptbk.exclude

upt/tarfiles

upt/gmatlogs

upt/drv-jpeek-jpeek.ps

upt/drv-jpeek.3l

upt/BOOKFILES~

upt/ch*.ps.gz

upt/ch*.ps

upt/,*

upt/art/,*

After the script makes the tar file, it touches a timestamp file named ptbk.last. The next time the script runs, it uses find -newer ( Section 9.8) to get only the files that have been modified since the timestamp file was touched.

The script uses scp and ssh-agent to copy the archive without asking for a password. You could hack it to use another method. For instance, it could copy using rcp (Section 1.21) or simply copy the file to another system with

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When extracting files from a tar archive, it's handy to be able to use wildcards. You have to protect them (Section 27.12) from the shell, so that they are passed directly to tar.

In general, tar can't do wildcard matching on the filenames within an archive. There's a terribly ugly hack that you can use to select the files you want anyway. Try a command like this:

'...' Section 28.14

% tar xvf /dev/rst0 `tar tf /dev/rst0 | egrep  'lib/(foo|bar)'`

What you're doing here is using tar twice. tar t will print the names of all the files on the tape. The pattern supplied to egrep ( Section 13.4) selects the pathnames containg lib/foo or lib/bar, and the resulting filenames are passed to the first tar command, which actually extracts the files from the archive. Note that these patterns are regular expressions, not wildcards (Section 32.2).

Here's another subtle but important point. Because the regular expression patterns in the example above are not anchored with ^ or $ characters (Section 32.4), they can match anywhere in the file pathnames. So lib/(foo|bar) would match a pathname like lib/foo as well as a pathname like /usr/lib/glib/foo.h.

Go to http://examples.oreilly.com/upt3 for more information on: GNU tar

One of the many improvements in GNU tar is that it understands wildcards in the names of files within an archive. (Remember that because you want tar, not the shell, to see these wildcards, you have to quote (Section 27.12) the filename arguments that have wildcards.)

Unlike the examples in the previous section, GNU tar uses wildcards, not regular expressions (Section 32.2). Unlike shells, the wildcard expressions in GNU tar can match across slashes (/) in pathnames.

Here's a demonstration of using wildcards: we want to extract all subdirectories named editor. Command 1 shows how you'd do it in non-GNU

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One problem with most versions of tar: they can't change a file's pathname when restoring. Let's say that you put your home directory in an archive (tape or otherwise) with a command like this:

% tar c /home/mike

What will these files be named when you restore them, either on your own system or on some other system? They will have exactly the same pathnames they had originally. So if /home/mike already exists, it will be destroyed. There's no way to tell tar that it should be careful about overwriting files; there's no way to tell tar to put the files in some other directory when it takes them off the tape, etc. If you use absolute pathnames (Section 31.2) when you create a tape, you're stuck. If you use relative paths (Section 31.2) (for example, tar c .), you can restore the files in any directory you want.

This means that you should:

Avoid using absolute paths when you create an archive (see below).
Use tar t to see what files are on the tape before restoring the archive.
Use GNU tar. It strips the leading / by default when creating archives. (You can give it the -P option to make it store absolute pathnames.)

Rather than giving a command like

tar c

/home/mike

, do something like:

% cd /

% tar c home/mike

Or, even more elegant, use -C on the tar command line:

% tar c -C /home/mike .

This command tells tar to cd to the directory /home/mike before creating an archive of . (the current directory). If you want to archive several directories, you can use several -C options:

% tar c -C /home/mike ./docs  -C /home/susan ./test

This command archives mike's docs directory and susan's test directory. [Note that it uses the subdirectory names, as we did in the second-previous example. When the files are extracted, they'll be restored to separate subdirectories, instead of all being mixed into the same . (current) directory. — JP]

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

tar's command line is one of Unix's little mysteries. It's difficult to associate arguments with options. Let's say you want to specify the block size (b), the output file ( f ), and an "exclude" file (X ). Where do you put all this information? It's easy enough to stick the option letters into a lump and put them into a command (tar cXbf). But where do you put the block size, the name of the exclude file, and so on?

List any arguments that you need after the block of key letters. You must place the arguments in the same order as the key letters, as shown in Figure 38-1.

Figure 38-1: tar options and arguments

In this command, keepout goes with the X option, 20 goes with the b option, and archive.shar goes with the f option. If we put the options in a different order, we also have to put the arguments in a different order (see Figure 38-2).

Figure 38-2: The same command, rearranged

Note that the files you want to put on the tape (or the files you want to extract from the tape) always go at the end of the command. These are not arguments to c or X; they are part of the command itself.

The dump command and a few others work the same way.

GNU tar understands this traditional syntax as well as two syntaxes with separate options. For instance, the command line above could also be written in either of the following ways with GNU tar:

% tar -c -b 20 -X keepout -f archive.tar *.txt

% tar --create --block-size=20 --exclude-from=keepout \

    --file=archive.tar *.txt

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There was a time when people used to debate whether BSD tar (Section 38.2, Section 39.2) (tape archiver) or System V cpio (copy in/out) was the better file archive and backup program. At this point, though, no one ships out cpio archives over the Net (Section 1.21). tar is widespread, and there are free versions available, including GNU tar (Section 39.3).

There's still a good reason to use cpio: it's better at recovering backups from partially damaged media. If a block of your tape or disk archive goes bad, cpio can probably recover all files except the one with the bad block. A tar archive may not fare as well. Though we don't give it much air time in this book, here are a few cpio basics:

To write out an archive, use the -o option and redirect output either to a tape device or to an archive file. The list of files to be archived is often specified with find (Section 9.1), but it can be generated in other ways — cpio expects a list of filenames on its standard input. For example:
```
% find . -name "*.old" -print | cpio -ocBv > /dev/rst8

                  
```
or:
```
% find . -print | cpio -ocBv > mydir.cpio

                  
```
To read an archive in, use the -i option and redirect input from the file or tape drive containing the archive. The -d option is often important; it tells cpio to create directories as needed when copying files in. You can restore all files from the archive or specify a filename pattern (with wildcards quoted to protect them from the shell) to select only some of the files. For example, the following command restores from a tape drive all C source files:
```
% cpio -icdv "*.c" < /dev/rst8

                  
```
Subdirectories are created if needed (-d), and cpio will be verbose (-v), announcing the name of each file that it successfully reads in.
To copy an archive to another directory, use the -p option, followed by the name of the destination directory. (On some versions of cpio, this top-level destination directory must already exist.) For example, you could use the following command to copy the contents of the current directory (including all subdirectories) to another directory:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This book mostly focuses on tools like tar, because that's what we expect most of you to use most of the time. However, there are other tools that are very important for large-scale backups that it's good to know at least a little about.

dump is an old Unix standby and a complete if somewhat arcane tool for backing up file systems. It is extremely useful for system administrators and personal machines, and it is available as part of the operating system on nearly any Unix. For industrial-strength backups, no simple solution beats dump — it is the most reliable tool for ensuring data consistency and stability. It's also a pain to use, so generally system administrators end up writing scripts around it to make it easier, or using a system like Amanda (see below).

The Advanced Maryland Automatic Network Disk Archiver, known as Amanda, is a free system for performing regular backups of one or more network-connected machines. Information on Amanda is generally available at http://www.amanda.org. Amanda uses tar or dump to do the actual work of backing up files; its job is to coordinate backups of multiple filesystems to one or more network-accessible tape drives on a regular basis.

Note also that full-scale backup processes need to address things such as tape lifetimes, electronic and physical security of backed-up data, off-site storage, incremental backup schemes and the like. Should you be in a position to need to set up such a process, read one of the good books on the subject — we might recommend O'Reilly's Unix Backup and Recovery.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The worst part of living in a nice big house is the headache of moving. The more stuff you've got room for, the more trouble it is to pack it up and take it with you.

The Unix operating system is a little bit like that. One of its real advantages is a filesystem that lets you organize your personal files into a hierarchical directory tree just like the much bigger tree that encompasses the entire filesystem. You can squirrel away all kinds of useful information into neat pigeonholes.

While your personal directory hierarchy is usually only two or three levels deep, for all practical purposes it can have as many levels as you like. And, as is true of any powerful and flexible tool, problems lie in wait for the sorcerer's apprentice. Directories and files grow increasingly complex the longer you use the system, with more forgotten files and more detailed organization.

This chapter will tackle the problems that can arise when you want to move a block of files (in one or many directories) from one place to another.

Maybe you're writing the files to a tape for safety (Section 38.3). In many cases though, this is a "backup and restore" problem. For example, if you were moving your account to another system, you might just ask the system administrator (if there is one) to archive your files to tape or floppy and restore them in the new location. Many new users are less aware that you can use the backup program tar ( Section 38.2) to create online archives that you can move from one place to another.

This situation is most likely to arise in a networked environment. You might be packaging files to ship as a package to another user. The files might be going to Usenet or an archive site on the Internet, for distribution to many users. Whether you're distributing an archive to lots of people or using it for yourself, though, most of the topics we cover in this chapter will apply.

— TOR

tar (Section 38.2) is a general-purpose archiving utility capable of packing many files into a single archive file, retaining information such as file permissions and ownership. The name

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The worst part of living in a nice big house is the headache of moving. The more stuff you've got room for, the more trouble it is to pack it up and take it with you.

The Unix operating system is a little bit like that. One of its real advantages is a filesystem that lets you organize your personal files into a hierarchical directory tree just like the much bigger tree that encompasses the entire filesystem. You can squirrel away all kinds of useful information into neat pigeonholes.

While your personal directory hierarchy is usually only two or three levels deep, for all practical purposes it can have as many levels as you like. And, as is true of any powerful and flexible tool, problems lie in wait for the sorcerer's apprentice. Directories and files grow increasingly complex the longer you use the system, with more forgotten files and more detailed organization.

This chapter will tackle the problems that can arise when you want to move a block of files (in one or many directories) from one place to another.

Maybe you're writing the files to a tape for safety (Section 38.3). In many cases though, this is a "backup and restore" problem. For example, if you were moving your account to another system, you might just ask the system administrator (if there is one) to archive your files to tape or floppy and restore them in the new location. Many new users are less aware that you can use the backup program tar ( Section 38.2) to create online archives that you can move from one place to another.

This situation is most likely to arise in a networked environment. You might be packaging files to ship as a package to another user. The files might be going to Usenet or an archive site on the Internet, for distribution to many users. Whether you're distributing an archive to lots of people or using it for yourself, though, most of the topics we cover in this chapter will apply.

— TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

tar (Section 38.2) is a general-purpose archiving utility capable of packing many files into a single archive file, retaining information such as file permissions and ownership. The name tar stands for tape archive, because the tool was originally used to archive files as backups on tape. However, use of tar is not at all restricted to making tape backups, as we'll see.

The format of the tar command is:

tar functionoptions 

               files...

where function is a single letter indicating the operation to perform, options is a list of (single-letter) options to that function, and files is the list of files to pack or unpack in an archive. (Note that function is not separated from options by any space.)

function can be one of:

c: Create a new archive.
x: Extract files from an archive.
t: List the contents of an archive.
r: Append files to the end of an archive.
u: Update files that are newer than those in the archive.
d: Compare files in the archive to those in the filesystem.

The most commonly used functions are c reate, extract, and table-of-contents.

The most common options are:

v: Prints verbose information when packing or unpacking archives. This makes tar show the files it is archiving or restoring. It is good practice to use this option so that you can see what actually happens, though if you're using tar in a shell script you might skip it so as to avoid spamming the user of your script.
k: Keeps any existing files when extracting — that is, prevents overwriting any existing files contained within the tar file.
f filename: Specifies that the tar file to be read or written is

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: tar

GNU tar has plenty of features; some people would say "too many." I don't agree. GNU tar has features I wish I'd had for years in more "standard" versions. This article lists my favorites. For a complete list, check the info documentation for tar.

Section 15.7 describes how to compress an archive file you've created. If you're using GNU tar, this is even easier, since tar itself can do the compression. Simply use the z option when writing or reading archives. For example, to make the gzipped tar archive progs.tar.gz from all ".c" and ".h" files:
```
% tar cvzf progs.tar.gz *.c *.h

                  
```
You can also use the long option - -gzip to get gzip compression, and the long option - -bzip2 to get bzip2 compression.
I've made the classic mistake of archiving files with their absolute pathnames (Section 38.11). GNU tar saves you from that goof. It always stores absolute pathnames as relative paths unless you add the - -absolute-names option.
Often I want to make a tape backup of my most recent work on a big project, but not all the thousands of files in a directory tree. The clumsy way to do that is by using find -mtime to make an include-file for the standard tar -I option. GNU tar to the rescue: its - -after-date option lets me tell it what directories to look in and how recently the files should have been changed.
When I extract an archive, I may be writing into a directory that has other files. The - -keep-old-files option tells GNU tar not to overwrite existing files.

One caution about GNU tar: it creates ANSI-format tar archives. Extracting one of these archives with the old V7 tar can cause warning messages like "tar: unexpected EOF." But, of course, GNU tar has an option to create old-format archives: - -old-archive.

—JP and TOR

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

How many times have you wished that you could get a copy of a file the way it looked an hour ago, or yesterday, or last year? That includes times when you just deleted the file — and, especially, when the file is too new for your computer's backup system to have made any copies of it. (You do have regular backups of your system, don't you? ;-)) RCS (Revision Control System) and CVS (Concurrent Version System) let you recover a previous version of a file from an archive. Many systems come with either RCS, CVS, or both installed already; if they don't appear to be on your system either install the appropriate package or grab the most current versions from FSF's website (http://www.fsf.org).

How does the archive get there? As you work, you periodically put a "snapshot" of the file into the archive. (The archive systems save the changes — not the whole file — so this doesn't take as much disk space as it might.) The archive remembers the date and time you store each version. You can enter a log message to describe what's changed since the last time you archived the file. You can do much more, but those are the basics.

When you need a previous version of the file, you read the archive log to decide which version is best (by date and time or by the log message). Then you use one command to get back that version. You don't have to wait for the system manager to load a tape.

Of course, these tools can't protect you from a disk crash or another disaster; that's what reliable backups are for. RCS and CVS are best for protecting you from accidentally deleting or corrupting files. But they're also great for group development projects: controlling who's working on a file, who did what when, and so on. That's especially true of CVS, which was designed to handle software developers from around the world collaborating on a project over a network — as well as a group of developers in the same office. One of my favorite features is the ability to see diff ( Section 11.1) listings of what's changed between versions.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Revision Control System (RCS) is a straightforward, file-based source-control system. It allows you to keep track of multiple snapshots or revisions of a file, so that you can back up to any previous version. It also allows you to note particular versions, so that you can do things such as reproduce the version of a file that you gave to someone else or released as part of a software release. Of course, it's useful for more than just software development; any time you want to change a file or set of files, revision control can be useful. To place a file under revision control using RCS:

% ci 

               filename

The ci (checkin) program will prompt you for a short description of the file and commit your changes. It will by default also delete the working copy; if you want to keep a read-only copy, use the -u (unlocked) option.

To then get a working copy of the file from scratch:

% co 

               filename

% co -l 

               filename

The co (checkout) command will get a read-only copy of the file from RCS. If you want to edit the file, use the co -l command (the option is a lowercase L and stands for lock). While you have the file checked out and locked, no one else can edit it. When you're done, return the file to RCS (check it in) using ci again. If you use the -l option to ci, it checks in your changes and checks out a new working copy, as if you did co -l again. When you check in the file, ci asks for a brief description of your changes. These can be very useful, later, to learn the history of revisions and to find a particular revision you might want to recover; the command rlog filename gives all of the stored change descriptions.

If you create a subdirectory called RCS in the directory where you keep the code or other text files you want to protect, the RCS files will be put there for you, rather than cluttering up your main directory.

It's a good idea (but not required) to add the characters

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: rcsrevs

The rcsrevs script tells you all the revision numbers that are stored in an RCS (Section 39.5) file. For instance:

% rcsrevs myprog

1.3

1.2

1.1

1.2.1.1

What good is that? Here are two examples.

rcsgrep -a ( Section 13.7) uses rcsrevs when it's searching all revisions of an RCS file. If you want to print all revisions, run a program across all revisions to do some kind of check, and so on, rcsrevs can give you the revision numbers to use in a loop (Section 28.9). The shell loop below gets all the revision numbers and stores them in the revnum shell variable one by one; it runs co -p (Section 39.5) to send each revision to the pr -h (Section 45.6) command for formatting with a custom header; the output of the commands in the loop goes to the printer.
'...' Section 28.14, > Section 27.12
```
$ for revnum in `rcsrevs 

                     somefile

                     `

> do

>   co -p -r$revnum 

                      | pr -h "

                     somefile

                      revision #$revnum"

> done | lpr

                  
```
You'd like to compare the two most recent revisions of several RCS files to see what the last change was, but the revision numbers in each file are different. (One file's latest revision might be 2.4, another file could be at 1.7, etc.) Use head (Section 12.12) to grab the two highest revision numbers from the rcsrevs output, tail -r (Section 12.9) to reverse the order (put the older revision number first), and sed to make the revision numbers into a pair of -r options (like -r1.6' -r1.7). Then run rcsdiff to do the comparisons and email (Section 1.21) them to bigboss:
? Section 28.12
```
% foreach file (*.cc *.h Makefile)

? set revs=`rcsrevs $f | head -2 | tail -r | sed 's/^/-r/'`
```

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Concurrent Version System, or CVS, is a version control system designed to support complex project structures or groups of people who are working together on a common set of files. Where RCS (Section 39.5) deals only with individual files, CVS allows you to work with entire projects as a whole. As we have mentioned before, while source control systems were originally developed primarily for use in developing software, they make a great deal of sense any time you want to keep track of changes to files. CVS is good for keeping track of changes to source files for a book or configuration files for qmail or apache, or for any number of other day-to-day tasks.

CVS stores its archives in a directory called a cvsroot. You tell CVS where to find the repository you want to use by setting the CVSROOT environment variable or using the -d option:

% setenv CVSROOT /home/cvsroot

% cvs checkout conf



% cvs -d /home/deb/cvs checkout book

Within a cvsroot are one or more repositories . Each repository is associated with a particular project (or in the case of a very complex project, a piece of a project). To work on a project, you much check out its repository to create a working area using cvs checkout, as in the example above. CVS is helpful and remembers which cvsroot you used for a particular checkout; future commands within that working area automatically use the right repository. For the record, the working area's cvsroot overrides the CVSROOT environment variable; the -d option overrides them both.

Once you have a working area, you have a writable copy of every file in that project. Edit to your heart's content. To incorporate changes made by other people, or see what you've changed, use cvs update:

% cd book

% cvs update

cvs update: Updating .

U ch18.sgm

M ch39.sgm

CVS update tells you a bit of information about each file that it touched or needs to touch. A U means that it updated your working copy from the repository; if you had also changed that file, it means that CVS successfully merged their changes with yours. A

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Here's a slightly more complex example of how to use CVS. I'm working on this book, via CVS, with my two main coauthors (who are on the east and west coasts of the United States). The repository, which has almost 1,000 files, is on a computer in the O'Reilly office in Massachusetts.

From the command line or in a shell setup file (Section 3.3), I need to set an environment variable (Section 35.3) named CVSROOT that tells CVS where the repository is and what my username is on that machine. In the C shell, for instance, I'd execute a command that sets my username to jpeek, the server hostname to bserver.east.oreilly.com, and the repository to /books/cvs. I'm also using ssh for secure access to the server, so I need to set the CVS_RSH environment variable and tell CVS to use the "ext" connection method:
```
setenv CVSROOT :ext:jpeek@bserver.east.oreilly.com:/books/cvs

setenv CVS_RSH ssh
```

I have a directory where I keep my local copies of the book files. To start, I check out my copy of the ulpt3 repository from the server:

!$ Section 30.3

% cd books

% cvs checkout ulpt3

cvs checkout: updating ulpt3

U ulpt3/0001.sgm

U ulpt3/0007.sgm

U ulpt3/0023.sgm

   ...more...

% cd !$

cd ulpt3

Now my ulpt3 subdirectory has the same files that the repository does. I can edit any of them, just as I'd edit files that aren't in CVS — but my changes don't make it back to the repository until I use the CVS command to do that.
Let's say I edit the file 0123.sgm. I'd like to write it back to the repository, where the other authors can grab it in case they're printing that part of the book. First I should update my workspace. This brings in any changes by other authors. If another author has updated 0123.sgm and put it in the archive before I do, CVS will merge the two files and expect me to resolve the differences:
```
% vi 0123.sgm

                     
```

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The location for certain types of installed files is very important. For instance, on many Unix systems, binary files accessible by users are located in the subdirectory /usr/bin or /usr/local/bin . If the applications aren't in these places, they may not be in the PATH environment variable and not easily accessible from the command line.

On my FreeBSD system, I've installed a utility called dos2unix , a file-formatting application that converts DOS newline character combinations to the Unix newline character. I used the FreeBSD Ports system to install the application, which automatically placed the program in my application directory, in my case /usr/local/bin. When I want to execute the application, I can run it from the command line without having to provide the location of the file:

dos2unix some.txt > new.txt

This command reformats the newline character of the contents of some.txt, converting DOS linebreaks to Unix ones.

The /usr/bin subdirectory differs from the /bin directory located directly off of the main root directory. The /bin directory has basic installed binaries built into the Unix operating system, with commands such as cd to change directory and so on. When you install an optional software application, it should not install software in the top-level binary subdirectory, but in /usr/bin, instead.

According to the Filesystem Hierarchy Standard (FHS), subdirectories (Linux- and BSD-specific) shown in Table 40-1 are located directly off the root directory within a standardized directory hierarchy.

Table 40-1: FHS root subdirectories
Subdirectory	Contents
`bin`	Application binaries

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The location for certain types of installed files is very important. For instance, on many Unix systems, binary files accessible by users are located in the subdirectory /usr/bin or /usr/local/bin . If the applications aren't in these places, they may not be in the PATH environment variable and not easily accessible from the command line.

On my FreeBSD system, I've installed a utility called dos2unix , a file-formatting application that converts DOS newline character combinations to the Unix newline character. I used the FreeBSD Ports system to install the application, which automatically placed the program in my application directory, in my case /usr/local/bin. When I want to execute the application, I can run it from the command line without having to provide the location of the file:

dos2unix some.txt > new.txt

This command reformats the newline character of the contents of some.txt, converting DOS linebreaks to Unix ones.

The /usr/bin subdirectory differs from the /bin directory located directly off of the main root directory. The /bin directory has basic installed binaries built into the Unix operating system, with commands such as cd to change directory and so on. When you install an optional software application, it should not install software in the top-level binary subdirectory, but in /usr/bin, instead.

According to the Filesystem Hierarchy Standard (FHS), subdirectories (Linux- and BSD-specific) shown in Table 40-1 are located directly off the root directory within a standardized directory hierarchy.

Table 40-1: FHS root subdirectories
Subdirectory	Contents
`bin`	Application binaries

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you've worked with multiple operating systems such as the Mac OS or Windows along with Unix, then you're aware that software installation on a Unix system — Solaris, Linux, Darwin, and so on — isn't necessarily as easy a task as it is on some of the other systems. The process can be difficult if you're installing open source code that you download from the Internet; many times open source code isn't packaged for ease in installation.

I've worked with Unix for years but still look at the process of installing a new piece of software as one would look at climbing a mountain: be prepared, be brave, and don't look back.

— SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many applications and utilities within the Unix environment come as source code that needs to be compiled and installed on your system. Because of this, the make utility is probably the most important utility you have within your Unix toolkit. However, the make utility installed on your system may not necessarily be compatible with the make utility used when the creators tested the software installation.

In fact, one of the problems that can cause the most problems with software installation is that the software compiles cleanly with GNU make but not with other versions of make, because different features of the installation process are supported with GNU make. This happens but not some of the older, more system-specific makes.

For instance, BSD operating systems such as FreeBSD and Darwin, as well as Solaris, have their own version of make in addition to accessibility to GNU make. In some of the systems, such as Darwin, GNU make is installed as the default. In others, such as FreeBSD, BSD make is the default. GNU make is installed but is usually called gmake. This typically isn't a problem because if the compilation fails, try gmake instead:

% gmake install

— SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

One of the problems associated with building and installing software within a Unix environment is ensuring that all the necessary libraries are present, the makefile is modified to fit the environment, and so on. The general building process is simplified by two GNU utilities: autoconf and automake.

The autoconf utility takes an input file called configure.in containing macros that determine how a configure file is built. The configure file, usually called Makefile.in , is then used by automake to create a Makefile that can be used to compile the application.

A README file should provide instructions on building an application, but if one is absent, you know that the application supports autoconf if you see a configure.in file, or see a script file called configure. If the package creator built a configure script manually, instructions will most likely be included within the README.

As a demonstration, I downloaded a GNU library called plotutils that provides graphics capability. After running gunzip and tar on the package to decompress the files, I looked at the topmost directory and found a configure file. I ran this using the following command:

> ./configure

The application can actually take a bit of time, and when finished, Makefiles have been generated for the application directories. All that's required at this point is to run make install as root:

> make install

Once the build was finished, I cleaned up by typing:

> make clean

> make distclean

The first make cleans up any in-process installation files; the second cleans up the distribution files.

The autoconf and automake utilities have greatly simplified installation of GNU and open source functionality.

— SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The dselect tool provides an easy-to-use, character-based graphical frontend for accessing dpkg (the traditional Debian installation package utility). To launch dselect, issue the command:

dselect

Figure 40-1 shows the screen that appears. The screen presents a simple menu with six items:

Access: Lets you choose the method used to access package files.
Update: Lets you update the list of available packages.
Select: Lets you choose packages for installation or removal.
Install: Initiates installation of selected packages.
Config: Initiates configuration of installed packages.
Remove: Initiates removal of packages selection for removal.
Quit: Exits dselect.

The menu items are generally used in the order in which they are presented.

Figure 40-1: The dselect screen

To choose the access method, use the arrow keys to highlight the Access menu item and press Enter. The screen shown in Figure 40-2 appears.

Figure 40-2: Choosing the access method

The most flexible access method — and the method that's generally recommended — is apt. Other available options include:

cdrom: Lets you install packages from a CD-ROM. This access method has been deprecated; you should use multi_cd instead.
multi_cd: Lets you install packages from a multivolume set of CD-ROMs.
nfs: Lets you install packages residing on an NFS server. This access method has been deprecated; you should use multi_nfs instead.
multi_nfs: Lets you install packages residing on an NFS server that has access to a multivolume set of packages.
harddisk: Lets you install packages residing on a hard disk partition that is not currently mounted. This access method has been deprecated; you should use apt or multi_mount

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The dselect program is useful, because it lets you browse a list of available packages, viewing their descriptions and dependencies, and selecting desired packages for installation. However, if you know the name of a package you want to install, apt-get is often the easiest way to install it. Before using apt-get, you must configure the sources.list file. This same file is used when you choose the apt access method of dselect. Even if you don't plan on using apt-get, you'll find the information in the following subsection useful.

The sources.list file resides in the /etc/apt directory. Like most other Linux configuration files, it can be revised by using an ordinary text editor, such as ae.

The file contains a series of lines, each specifying a source for packages. The lines are consulted serially, so it's usually advantageous to place lines that specify local sources — such as a CD-ROM — ahead of lines that specify remote sources. Doing so can save many minutes of download time.

Each line has the form:

deb uri distribution components

The uri is a universal resource identifier (URI) that specifies the computer on which the packages reside, the location of the packages, and the protocol used for accessing the packages. It has the following form:

protocol://host/path

Four protocols — sometimes called URI types — are recognized:

cdrom: A local CD-ROM drive
file: A directory of the local filesystem
http: A web server
ftp: An FTP server

The host part of the URI and the preceding pair of slashes (//) are used only for the http and ftp protocols. There, the host part of the URI gives the name of the host that contains the packages.

The path part of the URI always appears, with the preceding slash (/). It specifies the absolute path of the directory that contains the packages.

Here are some examples of typical URIs:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The GNU utility wget can be used to access files through the Internet using HTTP, HTTPS, or FTP. The best thing about the utility is that if the process is interrupted and started again, it continues from where it left off.

Go to http://examples.oreilly.com/upt3 for more information on: wget

The wget utility is installed by default in a lot of systems, but if you can't find it, it can be downloaded from GNU, at http://www.gnu.org/software/wget/wget.html.

The basic syntax for wget is very simple: type wget followed by the URL of the file or files you're trying to download:

wget http://www.somefile.com/somefile.htm

wget ftp://www.somefile.com/somefile

The file is downloaded and saved and a status is printed out to the screen:

--16:51:58--  http://dynamicearth.com:80/index.htm

           => `index.htm'

Connecting to dynamicearth.com:80... connected!

HTTP request sent, awaiting response... 200 OK

Length: 9,144 [text/html]



    0K -> ........                                               [100%]



16:51:58 (496.09 KB/s) - `index.htm' saved [9144/9144]

The default use of wget downloads the file into your current location. If the download is interrupted, by default wget does not resume at the point of interruption. You need to specify an option for this behavior. The wget options can be found in Table 40-2. Short and long forms of each option are specified, and options that don't require input can be grouped together:

> wget -drc URL

For those options that do require an input, you don't have to separate the option and the input with whitespace:

> wget -ooutput.file URL

Table 40-2: wget options
Option	Purpose	Examples
`-V`

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: curl

The cURL, or curl, application acts similar to wget, except that it works with more protocols, including FTPS, GOPHER, DICT, TELNET, FILE, and LDAP, as well as HTTP, HTTPS, and FTP (Section 40.8). It also supports kerberos, cookies, user authentication, file transfer resume and so on. You can access the application at http://curl.haxx.se, though curl is installed by default on some systems, including Mac OS X Darwin.

In particular, curl is used to download and install the GNU-Darwin auto-installer for OS X, otherwise known as One-Step.

The following command starts the process of installing the basefiles for One-Step. Note that the One-Step installation can take a considerable length of time, and you might get messages about needing to modify certain aspects of the installation, such as adding a font path and so on. Still, the instructions are very clear and once the installation is finished, you'll then be able to use One-Step.

# curl http://gnu-darwin.sourceforge.net/one_stop | csh

You can CTRL-c at any time during the download portion, and continue the installation at a later time. Use caution, though, with interrupting the installation during the final build portion. You can view the installation script for One-Step at http://gnu-darwin.sourceforge.net/one_stop/.

One of the applications installed during the process is pkg_add, which you can use to add additional packages at a later time by specifying the command followed by the URL of the package:

# pkg_add url

— SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The FreeBSD operating system has a terrific software installation system known as the FreePSD Ports. You can download the entire distribution collection as root giving a specific command within the /usr/ports directory:

# /stand/sysinstall

Once the Ports collection is installed, you can then easily install software by changing directory to the specific application and typing make install:

# cd /usr/ports

# cd lang

# cd ruby

# make install

# make clean

# make distclean

Not only will the Ports application install the application — in this case support for the Ruby programming language — it also pulls in any dependent files that might not exist on the system.

You may have problems downloading the distribution file because your system setup may not allow you to write to the /usr/ports/distfiles subdirectory for some reason. To install the distribution to a different location, set the DISTDIR environment variable to a different location:

# make DISTDIR=/local/dir/with/write/permission install

To removed an installed application, again change to the ports subdirectory of the application and type:

# make deinstall

— SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Instead of using the Ports you can install applications individually as packages, using the pkg_add utility, similar to that shown in (40.08).

To install using pkg_add, download the package by ftp'ing to the FreeBSD FTP server at ftp2.FreeBSD.org, and then change to the /pub/ports/packages directory. At that point, the directory at the FTP server should be similar to the Ports collection directory organization. Change directory to the category (such as "lang"). Get the gzipped tar (Section 15.7) file of the package for your application, using binary transfer. For instance, the Ruby scripting language interpretor is ruby-1.7.2.2002.05.23.tgz, which is then downloaded:

ftp > get /pub/ports/packages/lang/ruby-1.7.2.2002.05.23.tgz

Once the file is downloaded, type pkg_add and the package name:

# pkg_add ruby-1.7.2.2002.05.23.tgz

Instead of downloading the file manually, you can use the -r option to have the pkg_add application look for the latest build of an application and automatically download necessary dependent files and the target application itself:

# pkg_add -r ruby-1.7.2.2002.05.23.tgz

Note, though, that the newest versions of an application may not be in the stable build directory. If you want an application not on this tree, you'll have to download the application file manually.

— SP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: rpm

A popular tool used to find and install software packages — particularly in Linux — is RPM (at http://www.rpm.org/). In addition to working with Linux, RPM also works with Solaris, HP-UX, FreeBSD, NetBSD, and other systems.

To use RPM to install software, just type the following command:

rpm -i application.rpm

The -i option flags RPM to install the package. To uninstall the application, use:

rpm -e application.rpm

To upgrade a package, use the -U option:

rpm -U application.rpm

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Perl is an ecclectic, interpreted language with deep roots in Unix. It was originally written by Larry Wall, creator of other Unix staples such as patch and rn, to help with system administration tasks. Because many of its variables are prefixed with $, Perl often looks like an awk program or even a Bourne shell script. Like all appearances, this too can be deceiving. Perl is a complete programming language that supports both structured and object oriented programming. Getting started with Perl is easy, since many of the Bourne shell tricks you've seen will work (after a fashion) under Perl. As your knowledge grows, you'll find that Perl will help you scratch increasingly obscure itches. Because Perl has been ported to many different platforms, it brings a Unix-like API to whichever operating system is hosting it. Perl makes cross-platform programming a reality.

The complete guide to Perl is O'Reilly's Programming Perl, a book that weighs in at over 1500 pages. Therefore, only the barest of essentials can be presented here to help you identify your Perl installation, tinker with existing scripts, and install new modules. Luckily, Perl always comes with documentation that can be accessed through the perldoc (Section 41.10) system.

— JJ

Go to http://examples.oreilly.com/upt3 for more information on: perl

Before presenting the details of Perl syntax, it would be prudent to check whether or not Perl is on your system and learn how to install it if it isn't. Perl is an interpreted language whose interpreter is called perl. It is this program that reads, compiles and runs Perl source code. Normally, perl will be in your shell's path. It can often be found lurking in /usr/bin or /usr/local/bin. Use your system's find or locate command to track down perl if it doesn't appear in your command path. To see what version of Perl you have, use the -v flag like this:

$ perl -v



This is perl, v5.6.1 built for i686-linux



Copyright 1987-2001, Larry Wall



Perl may be copied only under the terms of either the Artistic License or the

GNU General Public License, which may be found in the Perl 5 source kit.



Complete documentation for Perl, including FAQ lists, should be found on

this system using `man perl' or `perldoc perl'.  If you have access to the

Internet, point your browser at http://www.perl.com/, the Perl Home Page.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Perl is an ecclectic, interpreted language with deep roots in Unix. It was originally written by Larry Wall, creator of other Unix staples such as patch and rn, to help with system administration tasks. Because many of its variables are prefixed with $, Perl often looks like an awk program or even a Bourne shell script. Like all appearances, this too can be deceiving. Perl is a complete programming language that supports both structured and object oriented programming. Getting started with Perl is easy, since many of the Bourne shell tricks you've seen will work (after a fashion) under Perl. As your knowledge grows, you'll find that Perl will help you scratch increasingly obscure itches. Because Perl has been ported to many different platforms, it brings a Unix-like API to whichever operating system is hosting it. Perl makes cross-platform programming a reality.

The complete guide to Perl is O'Reilly's Programming Perl, a book that weighs in at over 1500 pages. Therefore, only the barest of essentials can be presented here to help you identify your Perl installation, tinker with existing scripts, and install new modules. Luckily, Perl always comes with documentation that can be accessed through the perldoc (Section 41.10) system.

— JJ

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: perl

Before presenting the details of Perl syntax, it would be prudent to check whether or not Perl is on your system and learn how to install it if it isn't. Perl is an interpreted language whose interpreter is called perl. It is this program that reads, compiles and runs Perl source code. Normally, perl will be in your shell's path. It can often be found lurking in /usr/bin or /usr/local/bin. Use your system's find or locate command to track down perl if it doesn't appear in your command path. To see what version of Perl you have, use the -v flag like this:

$ perl -v



This is perl, v5.6.1 built for i686-linux



Copyright 1987-2001, Larry Wall



Perl may be copied only under the terms of either the Artistic License or the

GNU General Public License, which may be found in the Perl 5 source kit.



Complete documentation for Perl, including FAQ lists, should be found on

this system using `man perl' or `perldoc perl'.  If you have access to the

Internet, point your browser at http://www.perl.com/, the Perl Home Page.

This Perl is the latest stable version, 5.6.1. Perl is under very active development and newer versions may soon be available. As with all software projects, there is an unstable, developer's version of Perl that currently is 5.7.3. The version number scheme follows the pattern:

Revision number: These change only when the language is substantially redefined.
Version number: Even numbers indicate a stable, production-quality release. Odd numbers should only be used by Perl developers and the curious. Version numbers indicate an important change in the language that may affect scripts written to run under a previous version of Perl. Be sure to check out the perldelta manpage for details.
Subversion level: This number is better thought of as the patch level for a given version. Only bug fixes will appear with each new patch level release. of perl.

Local configuration information about perl can be obtained with the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you don't have Perl already or you'd like to install the latest version, you have a few options. The first is to get a precompiled version for your platform. This is an option of last resort, since you lose the opportunity to configure Perl for your system. Most Unix systems will compile the Perl source code cleanly.

To compile Perl, you will need to fetch the latest Perl source for the Comprehensive Perl Archive Network (CPAN) (Section 41.11). You can find the gzipped tar archive of the source code at http://www.cpan.org/src/stable.tar.gz. The archive is several megabytes, so those on a slow modem link need to plan accordingly. Unpack the archive with the following command:

$ gzip -dc stable.tar.gz | tar xvf -

You should now have a new subdirectory called perl-X.Y.Z (whatever the current version of Perl is). Change into this directory and you will be be ready to configure the build process for perl.

Like many Unix utilities, compiling Perl requires configuring a Makefile and then executing make . The Perl source comes with a robust Configure shell script that will prompt you to confirm information it finds about your system. Often, all the defaults are fine so you can tell the Configure not to prompt you for confirmation by passing the -de flag. If all goes well with the configuration stage, you'll want to start compiling the source with make. These steps can be effectively combined into to following idiom:

$ ./Configure -de && make test

Recall that the double ampersand is a kind of flow control operator in the shell that allows the make to happen only if the Configure succeeds. The Perl source comes with a test suite that attempts to verify that the build went according to plan. Since the test suite needs perl to be built, this command is similiar to typing:

$ ./Configure -de && make && make test

The configuration stage may report missing libraries (like those needed to make NDBM files or read shadowed password files). Generally, these messages are harmless. If an important dependency is missing, the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It is impossible to present a complete guide to programming Perl in this one small section, but you can glean enough information here to be able to modify existing Perl scripts and evaluate whether you'd like to learn more about this incredibly handy language.

Perl scripts bare a passing resemblence to Bourne shell scripts. Example 41-1 a script called writewav.pl that comes with the Perl module Audio::SoundFile. It converts a given sound file into WAV format. The details of what it's doing aren't important, but it does demonstrate some common Perl structures that you should understand at a high level.

Example 41-1. A sample Perl script

#!/usr/bin/perl -w



=head1 NAME



 writewav - Converts any sound file into .wav format



=cut



use Audio::SoundFile;

use Audio::SoundFile::Header;



my ($buffer, $length, $header, $reader, $writer);

my $BUFFSIZE = 16384;

my $ifile = shift @ARGV || usage( );

my $ofile = shift @ARGV || usage( );



$reader = Audio::SoundFile::Reader->new($ifile, \$header);

$header->{format} = SF_FORMAT_WAV | SF_FORMAT_PCM;

$writer = Audio::SoundFile::Writer->new($ofile,  $header);



while ($length = $reader->bread_pdl(\$buffer, $BUFFSIZE)) {

    $writer->bwrite_pdl($buffer);

}



$reader->close;

$writer->close;



sub usage {

  print "usage: $0 <infile> <outfile>\n";

  exit(1);

}

The first line of Example 41-1 should be familiar to shell hackers; it's the shebang line. When the first two bytes of a file are the characters #!, the shell uses the rest of that file's first line to determine which program should be used to interpret the rest of the file. In this case, the path to the Perl interpreter is given. Command line arguments can be given to the interpreter. Here -w instructs Perl to print warning messages when it finds code that is likely to be incorrect. This includes such common gaffes as trying to write to a read-only file handle, subroutines that recurse more than 100 times, and attempts to get the value of a scalar variable that hasn't been assigned a value yet. This flag is a new Perl programmer's best friend and should be used in all programs.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Data types are the kinds of values Perl supports. Common data types include arbitrarily long strings (e.g.,

"hi,

bob"

), intergers (e.g., 42) and floating point numbers (e.g., 3.14). Perl is a loosely typed language, which means that Perl works hard to let you forget about what kind of data you're dealing with. For the most part, you will be dealing with strings, which plays to Perl's strengths. To manipulate data, variables are employed. Table 41-1 lists the most common variable types in Perl. For the full story on Perl data types, read the perldata manpage.

Table 41-1: Common Perl variables
Name	Example	Description
`scalar`	`$lastname`, `$PI`	Holds single values
`array`	`@people`, `$peple[0]`	Holds an ordered sequence of scalar values
`hash`	`%cgi_params`, `$cgi_params{'action'}`	Holds a set of key-value pairs

When you want to store single values, like any of those given in the previous paragraph, you will use a

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

To do any interesting stuff with data, Perl needs to be able to branch and loop. Perl supports the C-like if -then-else construct, as the following shows:

if ( $password eq 'secret' ) {

  print "Come on in\n";

} else {

  print "Incorrect password\n";

}

You can also invert simple tests that only have one statement in the then block.

print "Don't I know you?\n" if $user eq 'joe';

You can invert the logic of if by using unless :

print "Please supply command line arguments\n" unless @ARGV;

The print happens only if @ARGV is empty.

Sometimes you need to iterate through each element of a list. This can be done with the foreach loop:

foreach my $thing (@my_room) {

  print "dusting $thing\n";

  dust($thing);

}

A synonym for foreach is for . Bourne shell hackers (or those who don't like typing) may feel more comfortable using for rather than then foreach.

Each time through the loop, $thing is aliased to the next element in @my_room. Any change to $thing will change that element in the array, so be careful. If you don't supply a scalar variable like $thing, Perl will set $_ for you each time through the loop. The previous example could also be written:

foreach (@my_room) {

  print "dusting $_\n";

  dust($_);

}

Sometimes you need to continue looping while an event is happening, like reading input from standard input:

while ( my $line = <STDIN> ) {

  print "I got: $line";

}

Each line of input a user provides is stored in $line, including the newline at the end. When the user hits the end-of-file control key (CTRL-D), the loop exits. Like the foreach loop, you can leave off the scalar variable while reading from a filehandle, and $_ will be set to the next line of input each time through the loop.

while (<>) {

  print "I got: $_";

}

Sometimes you need to interrupt the execute flow of your loop. Perl gives you three operators to do that (see Table 41-7).

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Perl is excellent at finding patterns in text. It does this with regular expressions, similar to the ones used by grep and awk. Any scalar can be matched against a regular expression with the matching binding operator, =~. For example:

if( $user =~ /jjohn/ ){

  print "I know you";

}

Without the matching binding operator, regular expressions match against the current value of $_. For example:

while (<>) {

  if (/quit/i) {

    print "Looks like you want out.\n";

    last;

  }

}

In this code, each line of input is examined for the character sequence quit. The /i modifier at the end of the regular expression makes the matching case-insensitive (i.e., Quit matches as well as qUIT).

As with regular expressions in other utilities, Perl attempts to find the leftmost and longest match for your pattern against a given string. Patterns are made up of characters (which normally match themselves) and special metacharacters, including those found in Table 41-8.

Table 41-8: Common Perl regular expression metacharacters
Operator	Description
`^`	Pattern must match at the beginning of the line.
`$`	Pattern must match at the end of the line.
`.`	Match any character (expect the newline).
pat1

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are many built-in Perl operators that mimic Unix system calls and standard utilities, as are partially listed in Table 41-9. Those that aren't hardcoded into the language are often available through modules (Section 41.9). In fact, there are too many Unix-like operators to describe here, but this sampling should give you a good start.

Table 41-9: Perl filesystem functions
Function	Example	Description
`chmod`	`chmod 0775,` filenames	Change file permission on given file or list of files; same as the system command.
`chown`	`chown` UID `,` GID `,` filenames	Change owner and group on given list of filenames; same as the system command.
`rename`	`rename` oldname `,` newname

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Modules are Perl's way of extending functionality, in the same way C has library files. Modules can be used to encapsulate a set of related function calls (the way Data::Dumper does), implement pragmas (like use strict), or create object classes (like HTML::TokeParser). Whatever a module does, it must first be installed on your system (Section 41.11) before you can use it.

Using a module in Perl is often straightforward. For example, the Data::Dumper module has a function called Dumper that takes a reference to a variable and deconstructs the entire structure into a printable string. This is an invaluable debugging tool. The following code shows Data::Dumper in action:

use Data::Dumper;

print "The current environment is: ", Dumper(\%ENV), "\n";

An abbreviated version of the output from this code is this:

The current enviroment is: $VAR1 = {

          'DISPLAY' => ':0',

          'COLORTERM' => 'gnome-terminal',

          'QTDIR' => '/usr/lib/qt-2.3.0',

          'PVM_RSH' => '/usr/bin/rsh',

          'OSTYPE' => 'linux-gnu',

          'PWD' => '/home/jjohn/docs/unix_powertools/upt',

          'EDITOR' => 'emacs -nw',

          'LOGNAME' => 'jjohn',

          'MACHTYPE' => 'i386-redhat-linux-gnu',

          'SHELL' => '/bin/bash',

          'MAIL' => '/var/spool/mail/jjohn',

          '_' => '/usr/local/bin/perl',

          'HISTSIZE' => '1000',

          'CVS_RSH' => 'ssh1',

          'HOSTNAME' => 'marian',

          'TERM' => 'xterm',

          ...

        };

In this code, the Data::Dumper is made available to your script with the use statement. You should be aware that use happens at the script's compile time, meaning that you can't use this statement to dynamically load modules at runtime (but this is possible; see Programming Perl for details). Data::Dumper automatically makes the function Dumper available to your script. Here the global hash %ENV, which contains all your shell's environment variables, is deconstructed. Dumper can take multiple variables, so when looking at a hash or array, be sure to prefix the variable with the reference operator (Section 41.5.4)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

We all need a little help sometimes, and it's at those times that perldoc comes in handy. Normally, core Perl and module documentation is accessible through your system's manpage system, but you can also use the perldoc program, which has a few convenient features that you should be aware of. Like man, perldoc takes the name of a module or core Perl document as an argument.

Your system's perl comes bundled with hundreds of pages of very readable documentation. The top of the document tree can be accessed with either perldoc perl or man perl. This page is little more than a table of contents for the rest of the perl documentation. There are over 40 documents listed there, but there are a couple that will be immediately useful to novice Perl programmers, as Table 41-10 shows.

Table 41-10: Frequently used Perl manpages
Name	Description
perlsyn	The complete guide to Perl syntax
perldata	Perl's data types explained
perlop	Perl's operators and their precedence
perlfunc	The complete guide to all of Perl's built-in functions
perlre	The complete guide to Perl's regular expressions

In many cases (such as the ones above), perldoc doesn't do anything man can't. However with perldoc, you can easily look up built-in Perl functions with the -f flag (-t formats any POD elements for a text console). For instance, to see the entry on print, try this:

$ perldoc -tf print

You'll get back something like the following (which has been abbreviated):

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Comprehensive Perl Archive Network (CPAN), whose URL is http://www.cpan.org, is the place to get modules, scripts, and the Perl source code. This system is mirrored all over the world, so consult http://www.cpan.org/SITES.html or http://mirror.cpan.org for the server nearest you. There is a really complete CPAN FAQ that can be found at http://www.cpan.org/misc/cpan-faq.html.

This section covers obtaining and installing modules from CPAN. If your installation of Perl is up to date, module installation is trivial. If you've got a "unique" system, you may need to take matters into your own hands.

In a fit of inspired genius (or madness), the CPAN module was created to automate the task of fetching and installing modules. If you want to install the Text::AutoFormat suite, it's as easy as becoming superuser on your system and typing:

# perl -MCPAN -e 'install Text::AutoFormat'

Perl has many command-line switches. Here, -M (equivalent to use module) and -e (execute the next argument as perl code) are used. If you've never run the CPAN module before, be prepared to answer a lot of questions about your network setup and where certain system binaries are. Luckily, you can usually accept the defaults safely. Once that's done, the CPAN module will go to the CPAN mirror you specified; find the latest version of the module you asked for; and download, unpack, configure, and install it for you with no additional typing. Now that's twenty-first-century library management! If your module depends on other modules not installed on your system, CPAN will attempt to fetch and install the missing modules. In fact, you can update the CPAN module itself with:

# perl -MCPAN -e 'install Bundle::CPAN'

The CPAN module also has an interactive shell you can access like this:

$ perl -MCPAN -e shell

Why bother with the interactive shell? Sometimes you want to install several unrelated modules at once. This is done more conveniently in the CPAN shell. Alternately, you may want only to download module archives without actualling installing them. The entire range of shell options can be found with the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

All of the various grep-like utilities perform pretty much the same function, with minor differences — they search for a specified pattern in some or all of a file and display that pattern with varying amounts of surrounding context.

As you use Unix more and more, you will find yourself wanting to do an increasing number of grep-like tasks, but no particular Unix utility will quite suit them all (hence the need for the various grep utilities discussed earlier). You'll start accumulating C programs, awk scripts, and shell scripts to do these different tasks, and you'll be craving one utility that can easily encompass them all so you don't have to waste the disk space for all of those binaries. That utility is Perl (Section 41.1), the "Practical Extraction and Report Language" developed by Larry Wall. According to the documentation accompanying Perl, it is "an interpreted language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information."

For example, to search for a pattern in the header of a Usenet message:

perl -ne 'exit if (/^$/); print if (/pattern/);' filename

[This works because mail and Usenet (Section 1.21) messages always use a blank line — indicated by ^$ in regular expression syntax — to separate the header from the body of the message. — TOR]

[The -n flag tells perl to wrap the contents of -e into the body of a while(<>){ ... } loop. — JJ]

To do a search for a pattern and print the paragraphs in which it appears:

perl -ne '$/ = "\n\n"; print if (/pattern/);' filename

[This assumes that paragraphs are delimited by a double linefeed — that is, a blank line. You'd have to adjust this script for a troff or T_EX document where paragraphs are separated by special codes. — TOR]

Searching through files is one of Perl's strengths, but certainly not its only strength. Perl encompasses all the functionality of sed, awk, grep, find, and other Unix utilities. Furthermore, a Perl program to do something originally done with one or more of these utilities is usually faster and easier to read than the non-Perl solution.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Because Perl supports Berkeley sockets, all kinds of networking tasks can be automated with Perl. Below are some common idioms to show you what is possible with Perl and a little elbow grease.

The suite of classes that handle all the aspects of HTTP are collectively known as LWP (for libwww-perl library). If your Perl installation doesn't currently have LWP, you can easily install it with the CPAN module (Section 41.11) like this:

# perl -MCPAN -e 'install Bundle::LWP'

If you also included an X widget library such as Tk, you could create a graphic web browser in Perl (an example of this comes with the Perl Tk library). However, you don't need all of that if you simply want to grab a file from a web server:

use LWP::Simple;

my $url = "http://slashdot.org/slashdot.rdf";

getstore($url, "s.rdf");

This example grabs the Rich Site Summary file from the popular tech news portal, Slashdot, and saves it to a local file called s.rdf. In fact, you don't even need to bother with a full-fledged script:

$ perl -MLWP::Simple -e 'getstore("http://slashdot.org/slashdot.rdf", "s.rdf")'

Sometimes you want to process a web page to extract information from it. Here, the title of the page given by the URL given on the command line is extracted and reported:

use LWP::Simple;

use HTML::TokeParser;



$url = $ARGV[0] || 'http://www.oreilly.com';

$content = get($url);

die "Can't fetch page: halting\n" unless $content;



$parser = HTML::TokeParser->new(\$content);

$parser->get_tag("title");

$title = $parser->get_token;

print $title->[1], "\n" if $title;

After bringing in the library to fetch the web page (LWP::Simple) and the one that can parse HTML (HTML::TokeParser), the command line is inspected for a user-supplied URL. If one isn't there, a default URL is used. The get function, imported implicitly from LWP::Simple, attempts to fetch the URL. If it succeeds, the whole page is kept in memory in the scalar $content. If the fetch fails, $content will be empty, and the script halts. If there's something to parse, a reference to the content is passed into the HTML::TokeParser object constructor. HTML::TokeParser deconstructs a page into individual HTML elements. Although this isn't the way most people think of HTML, it does make it easier for both computers and programmers to process web pages. Since nearly every web page has only one

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Python is an interpreted scripting language, much like Perl or Tcl. Python's primary focus is on clear, concise code, and it has a feature set and wide variety of available modules designed to support this goal. In many ways, Python is an extremely scalable language; complex systems can be relatively easily built in Python without losing maintainability. From the Python home page (http://www.python.org):

Python is an interpreted, interactive, object-oriented programming language. It is often compared to Tcl, Perl, Scheme or Java.
Python combines remarkable power with very clear syntax. It has modules, classes, exceptions, very high level dynamic data types, and dynamic typing. There are interfaces to many system calls and libraries, as well as to various windowing systems (X11, Motif, Tk, Mac, MFC). New built-in modules are easily written in C or C++. Python is also usable as an extension language for applications that need a programmable interface.
The Python implementation is portable: it runs on many brands of UNIX, on Windows, DOS, OS/2, Mac, Amiga... If your favorite system isn't listed here, it may still be supported, if there's a C compiler for it. Ask around on comp.lang.python — or just try compiling Python yourself.
Python is copyrighted but freely usable and distributable, even for commercial use.

— DJPH

Go to http://examples.oreilly.com/upt3 for more information on: python

Installing Python is generally very simple. Either install the appropriate binary package for your platform, or download the latest source from http://www.python.org. (Note that some Linux distributions include Python by default.) A source install is as simple as untarring the distribution, then running:

% ./configure

% make

% make install

You can run the Python interpreter interactively and find out what version you have and details about its compilation. As an example, on my laptop (which runs Windows but also has a Cygwin Unix-like environment installed), Python reports:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Python is an interpreted scripting language, much like Perl or Tcl. Python's primary focus is on clear, concise code, and it has a feature set and wide variety of available modules designed to support this goal. In many ways, Python is an extremely scalable language; complex systems can be relatively easily built in Python without losing maintainability. From the Python home page (http://www.python.org):

Python is an interpreted, interactive, object-oriented programming language. It is often compared to Tcl, Perl, Scheme or Java.
Python combines remarkable power with very clear syntax. It has modules, classes, exceptions, very high level dynamic data types, and dynamic typing. There are interfaces to many system calls and libraries, as well as to various windowing systems (X11, Motif, Tk, Mac, MFC). New built-in modules are easily written in C or C++. Python is also usable as an extension language for applications that need a programmable interface.
The Python implementation is portable: it runs on many brands of UNIX, on Windows, DOS, OS/2, Mac, Amiga... If your favorite system isn't listed here, it may still be supported, if there's a C compiler for it. Ask around on comp.lang.python — or just try compiling Python yourself.
Python is copyrighted but freely usable and distributable, even for commercial use.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: python

Installing Python is generally very simple. Either install the appropriate binary package for your platform, or download the latest source from http://www.python.org. (Note that some Linux distributions include Python by default.) A source install is as simple as untarring the distribution, then running:

% ./configure

% make

% make install

You can run the Python interpreter interactively and find out what version you have and details about its compilation. As an example, on my laptop (which runs Windows but also has a Cygwin Unix-like environment installed), Python reports:

% python

Python 2.2 (#1, Dec 31 2001, 15:21:18)

[GCC 2.95.3-5 (cygwin special)] on cygwin

Type "help", "copyright", "credits" or "license" for more information.

>>>

To see which modules are compiled into your version of Python, examine sys.builtin_module_names:

>>> import sys

>>> print sys.builtin_module_names

('_ _builtin_ _', '_ _main_ _', '_socket', '_sre', '_symtable', 'exceptions',

 'gc', 'imp', 'marshal', 'new', 'posix', 'signal', 'sys', 'xxsubtype')

These are just the modules that are an integral part of your version of the interpreter. For a complete list of modules installed in your Python, look in all of the directories listed in sys.path:

>>> print sys.path

['', '/usr/lib/python2.2', '/usr/lib/python2.2/plat-cygwin',

 '/usr/lib/python2.2/lib-tk', '/usr/lib/python2.2/lib-dynload',

 '/usr/lib/python2.2/site-packages']

Generally, checking the documentation for the version of Python you have will tell you which modules are normally installed; the site-packages directory is where further packages installed on your machine will likely have been installed.

There is a large repository of modules (and other Python code resources) for Python available at the Vaults of Parnassus (http://www.vex.net/parnassus/), which includes a search mechanism for finding what you're looking for. Most modules will use Distutils to package their distributions.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If you've written code in a procedural or functional language before, many parts of Python will seem familiar. Here's a quick overview of the flavor of the language. There is a lot of both reference and tutorial information available on the web (start at http://www.python.org) as well as in books like O'Reilly's Programming Python. In fact, much of the information in this chapter was gleaned or paraphrased from the official Python reference documentation.

The number one complaint of Python detractors is almost always its use of indentation as a significant part of its syntax. Most languages use begin/end tags or curly braces ({}) to mark blocks of code and have line termination punctuation (many use the semicolon (;) as a line termination marker). In Python, indentation is used to define blocks of code, and lines are terminated with a return. The actual amount of indentation within a block is arbitrary, but it must be consistent:

if a:

    statement1

    statement2    # Consistent indentation

else:

    statement3

      statement4  # Inconsistent indentation (error)

Python assumes eight-space tab characters. If you have your editor set to four-space tabs, for example, this can bite you if there are mixed spaces and tabs. Either use eight-space tabs, or stick to spaces.

Long statements can span multiple lines by using the backslash (\) to continue the line:

>>> a = math.cos( 3 * ( x - n ) ) + \

...     math.sin( 3 * ( y - n ) )

Lines that are already grouped within triple-quotes, parentheses (...), brackets [...], or braces {...} can span multiple lines without needing to use the backslash.

Python's indentation requirements take a little getting used to, but they guarantee a certain level of readability, and editors like vim and emacs can keep track of the details for you trivially. (vim has a syntax configuration for editing Python, and emacs and xemacs both have a python-mode specifically for dealing with Python.)

Both procedural and functional languages organize programs by dividing them into smaller units called

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Python has a number of core modules designed to deal with interacting with the web. Python can act as a web client, pulling down web resources and POSTing form results. Python has support for SSL connections in a reasonably transparent fashion. CGI scripts are easy to write in Python, and there is also an Apache module for running Python scripts within the webserver itself.

urllib ( Section 42.5) provides basic functions for opening and retrieving web resources via their URLs.
urllib2 ( Section 42.6) provides an extended, extensible interface for accessing web resources.
htmllib and HTMLParser (Section 42.7) provide the ability to parse HTML.
cgi (Section 42.8) provides functions for writing CGI scripts.
mod_python ( Section 42.9) is an Apache module for running Python within the Apache webserver, rather than seperately as with CGI scripts.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The application-level access to most web client activities is through modules called urllib and urllib2 (Section 42.6). urllib is the simple web interface; it provides basic functions for opening and retrieving web resources via their URLs.

The primary functions in urllib are urlopen( ), which opens an URL and returns a file-like object, and urlretrieve( ), which retrieves the entire web resource at the given URL. The file-like object returned by urlopen supports the following methods: read( ), readline( ), readlines( ), fileno( ), close( ), info( ), and geturl( ). The first five methods work just like their file counterparts. info( ) returns a mimetools.Message object, which for HTTP requests contains the HTTP headers associated with the URL.

geturl(

)

returns the real URL of the resource, since the client may have been redirected by the web server before getting the actual content.

urlretrieve( ) returns a tuple

(filename,

info)

, where filename is the local file to which the web resource was copied and info is the same as the return value from urlopen's info( ) method.

If the result from either urlopen( ) or urlretrieve( ) is HTML, you can use htmllib to parse it.

urllib also provides a function urlencode( ), which converts standard tuples or dictionaries into properly URL-encoded queries. Here is an example session that uses the GET method to retrieve a URL containing parameters:

>>> import urllib

>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})

>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)

>>> print f.read( )

The following example performs the same query but uses the POST method instead:

>>> import urllib

>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})

>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)

>>> print f.read( )

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

urllib2 provides an extended, extensible interface to web resources. urllib2's application-level interface is essentially identical to urllib's urlopen( ) function (Section 42.5). Underneath, however, urllib2 explicitly supports proxies, caching, basic and digest authentication, and so forth.

urllib2 uses an Opener, made up of a series of Handlers, to open a URL; if you know you want to use a particular set of features, you tell urllib2 which Handlers to use before you call urlopen( ). urllib2 is extensible largely because if you need to deal with some odd set of interactions, you can write a Handler object to deal with just those interactions and incorporate it into an Opener with existing Handlers. This allows you to deal with complex behavior by just combining very simple sets of code.

For example, to retrieve a web resource that requires basic authentication over a secure socket connection:

>>> import urllib2

>>> authHandler = urllib2.HTTPBasicAuthHandler( )

>>> authHandler.add_password("private, "https://www.domain.com/private",

...                          "user", "password")

>>> opener = urllib2.build_opener(authHandler)

>>> urllib2.install_opener(opener)

>>> resource = urllib2.urlopen("https://www.domain.com/private/foo.html")

>>> print resource.read( )

To implement a new Handler, you simply subclass from urllib2.BaseHandler and implement the methods appropriate to the behavior you want to handle.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Python provides the htmllib module for parsing HTML content, which is often useful when dealing with web resources. Python also has an HTMLParser module, which handles both XHTML and HTML and provides a slightly lower-level view of the content. HTMLParser is also slightly simpler to use, since htmllib uses sgmllib and thus understands many of the complexities of SGML.

HTMLParser provides a class that the user subclasses from, defining methods that are called as tags are found in the input. The example below is a very basic HTML parser that uses the HTMLParser.HTMLParser class to print out tags as they are encountered:

from HTMLParser import HTMLParser



class MyHTMLParser(HTMLParser):

    def handle_starttag(self, tag, attrs):

        print "Encountered the beginning of a %s tag" % tag

    def handle_endtag(self, tag):

        print "Encountered the end of a %s tag" % tag

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Python provides the cgi module for writing CGI scripts. Much of the grunt work of writing a CGI script is in dealing with parsing the parameters handed to the script by the web server. The cgi module deals with all of those details and more.

To use the cgi module, use import cgi rather than from cgi import*. The cgi module defines a lot of symbols (many for backwards compatibility) that you don't want polluting your namespace.

When you write a new script, consider adding the line:

import cgitb; cgitb.enable( )

This activates a special exception handler that will display detailed reports in the web browser if any errors occur. If you'd rather not show the guts of your program to users of your script, you can have the reports saved to files instead, with a line like this:

import cgitb; cgitb.enable(display=0, logdir="/tmp")

It's very helpful to use this feature during script development. The reports produced by cgitb provide information that can save you a lot of time tracking down bugs. You can always remove the cgitb line later when you have tested your script and are confident that it works correctly.

To get to information submitted to the CGI script, instantiate a FieldStorage object:

form = cgi.FieldStorage( )

The FieldStorage object acts much like a dictionary of CGI information; it implements the methods has_key( ) and keys( ) and can be accessed using the [ ] operator. For instance, the following code (which assumes that the Content-Type: header and blank line have already been printed) checks that the fields name and addr are both set to a non-empty string:

form = cgi.FieldStorage( )

if not (form.has_key("name") and form.has_key("addr")):

    print "<H1>Error</H1>"

    print "Please fill in the Name and Address fields."

    return

print "<p>Name: %s</p>" % form["name"].value

print "<p>Address: %s</p>" % form["addr"].value

...further form processing here...

The cgi module also supports ways to deal with multiple-selection form elements and uploaded files.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

mod_python is an Apache module for running Python within the Apache webserver. It's much faster than CGI scripts and generally uses less resources overall. mod_python also allows advanced functionality such as maintaining persistent database connections across web requests and access to internal Apache APIs. Information on mod_python and distributions are available at http://www.modpython.org.

Apache's basic methodology for handling web requests is to deal with them in phases. There is a phase for each significant element of handling the request, including authentication, content generation, and logging. Apache modules can provide a seperate handler for each phase; mod_python simply allows you to write those handlers in Python. This allows complete control over everything Apache does to process a request.

A mod_python handler is a function that takes the Request object as an argument; a Request represents a single web request and contains all the information Apache knows about that request (requested URL, method, headers, and so forth).

Each phase's handler has a specific name that Apache recognizes (and uses in its configuration file): PythonHandler, PythonAuthenHandler, PythonLogHandler and so forth. Most mod_python scripts need to implement only the main handler, PythonHandler.

mod_python finds the appropriate function to call by dropping the leading Python from the handler name, and using an all-lowercase function name. Thus, most mod_python scripts will look something like this:


from mod_python import apache



def handler(request):

    request.content_type = "text/plain"

    request.send_http_header( )

    request.write("Hello World!")

    return apache.OK

This handler simply imports the apache API and then responds to every request with a plain text

Hello

World!

. It returns apache.OK to tell Apache that the request was successful.

For more information on dealing with mod_python, read the documentation.

One gotcha: mod_python's way of installing a mod_python handler is a little counterintuitive due to the way Apache handlers work. Make sure you understand how

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Comparing languages can generate a lot of heat and very little light. However, "Why not just use Perl?" is such a common question that I'll try to provide at least a basic understanding of the relative strengths and weaknesses of Python versus Perl. Remember that you can write good code or bad code in pretty much any language, but understanding whether your tool is best at driving nails or screws is always useful.

Perl's driving motto is "There's more than one way to do it." Because of this priority and the huge archive of Perl modules on CPAN, Perl is an incredibly useful tool for building quick one-off scripts or hacking together tools in a very short time. However, it also means that it's very easy to write Perl code that will be impenetrable six months down the road. Perl provides very little assistance to someone who wants to write complex systems clearly. Features like perl -w (warnings), use strict, and Perl's module support help maintainability, but it still requires a great deal of care and discipline.

Python's support for maintainability, on the other hand, is excellent. Python's rich collection of modules and the fact that it's an interpreted language allow relatively fast development, if not quite as fast as in Perl. Generally, the more complex the system you're trying to build and the longer you expect to use it, the more potential there is for gain in using Python over Perl.

Personally, when tossing together quick one-offs or scripts that are very regular expression-heavy, I use Perl. Perl's regular expression support is so fundamental to the language that it's worth it, and its Swiss-Army-knife nature is perfect for things I don't expect to need again later. I also tend to use Perl when I want to write a very portable script, as most Unixes include Perl as part of the base system these days, whereas Python, while just as portable, tends to need to be installed seperately. When I want to build more complex scripts or larger systems, and maintainability is thus a higher priority, I use Python. I often use Python even for smaller things if I intend to keep them around for a while.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There is basically no difference between reading data from a file and reading data from a terminal. Likewise, if a program's output consists entirely of alphanumeric characters and punctuation, there is no difference between writing to a file, writing to a terminal, and writing to the input of another program (as in a pipe).

The standard I/O facility provides some simple defaults for managing input/output. There are three default I/O streams: standard input, standard output, and standard error. By convention, standard output (abbreviated stdout) consists of all "normal" output from your program, while standard error (stderr) consists of error messages. It is often a convenience to be able to handle error messages and standard output separately. If you don't do anything special, programs will read standard input from your keyboard, and they will send standard output and standard error to your terminal's display.

Standard input (stdin) normally comes from your keyboard. Many programs ignore stdin; you name files directly on their command line — for instance, the command cat file1 file2 never reads its standard input; it reads the files directly. But without filenames on the command line, Unix commands that need input will usually read stdin. Standard input normally comes from your keyboard, but the shell can redirect stdin from a file. This is handy for Unix commands that can't open files directly — for instance, mail ( Section 1.21). To mail a file to joan, use < filename — to tell the shell to attach the file, instead of your keyboard, to mail's standard input:

% mail joan < myfile

The real virtue of standard I/O is that it allows you to redirect input or output away from your terminal to a file. As we said, Unix is file-based (Section 1.19). Because terminals and other I/O devices are treated as files, a program doesn't even need to know if it is sending its output to a terminal or to a file. For example, if you want to run the command

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There is basically no difference between reading data from a file and reading data from a terminal. Likewise, if a program's output consists entirely of alphanumeric characters and punctuation, there is no difference between writing to a file, writing to a terminal, and writing to the input of another program (as in a pipe).

The standard I/O facility provides some simple defaults for managing input/output. There are three default I/O streams: standard input, standard output, and standard error. By convention, standard output (abbreviated stdout) consists of all "normal" output from your program, while standard error (stderr) consists of error messages. It is often a convenience to be able to handle error messages and standard output separately. If you don't do anything special, programs will read standard input from your keyboard, and they will send standard output and standard error to your terminal's display.

Standard input (stdin) normally comes from your keyboard. Many programs ignore stdin; you name files directly on their command line — for instance, the command cat file1 file2 never reads its standard input; it reads the files directly. But without filenames on the command line, Unix commands that need input will usually read stdin. Standard input normally comes from your keyboard, but the shell can redirect stdin from a file. This is handy for Unix commands that can't open files directly — for instance, mail ( Section 1.21). To mail a file to joan, use < filename — to tell the shell to attach the file, instead of your keyboard, to mail's standard input:

% mail joan < myfile

The real virtue of standard I/O is that it allows you to redirect input or output away from your terminal to a file. As we said, Unix is file-based (Section 1.19). Because terminals and other I/O devices are treated as files, a program doesn't even need to know if it is sending its output to a terminal or to a file. For example, if you want to run the command cat file1 file2

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

What's wrong with this command line?

cat Section 12.2

% cat filename | tr -d '\015' > newfile

As Tom Christiansen wrote in a Usenet article:

A wise man once said: if you find yourself calling cat with just one argument, then you're probably doing something you shouldn't.

The command line above only uses cat to feed the file to the standard input of tr. It's a lot more efficient to have the shell do the redirection for you with its < character (Section 43.1):

% tr -d '\015' < filename > newfile

—JP and TC

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A vertical bar character (|) on a command line pipes the standard output of a process to another process. How can you pipe the standard error but not the standard output? You might want to put a long-running cruncher command in the background, save the output to a file, and mail yourself a copy of the errors. In the C shell, run the command in a subshell (Section 43.7). The standard output of the command is redirected inside the subshell. All that's left outside the subshell is the standard error; the |& operator (Section 43.5) redirects it (along with the empty standard output) to the mail (Section 1.21) program:

% (cruncher > 

               outputfile

               ) |& mail  

               yourname &

[1] 12345

Of course, you don't need to put that job in the background. If you want the standard output to go to your terminal instead of a text file, use /dev/tty (Section 36.15) as the outputfile.

The Bourne shell gives you a lot more flexibility and lets you do just what you need. The disadvantage is the more complicated syntax (Section 36.16). Here's how to run your cruncher program, route the stderr through a pipe to the mail program, and leave stdout going to your screen:

$ (cruncher 3>&1 1>&2 2>&3 3>&-) | mail 

               yourname

                &

12345

If this example makes your head hurt a little, you're not alone. The key to understanding this arcana is to know that programs don't refer to files by name like users do. Instead, when a program wants to read or write to a file, it must ask the operating system for a file stream that has an integer file descriptor associated with it. Every program has three file streams opened by default: standard input, standard output, and standard error. The file descriptors associated with standard input and standard error are 1 and 2, respectively. These file streams may be duplicated; that is, the data stream pointed by the file descriptor on the left will now go to data stream pointed to by the file descriptor on the right. If you wanted to redirect both standard error and standard output to

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

If your window onto Unix (terminal, X window, communications program, whatever) doesn't have a way to show you the previous screenful, using a pager program like more , pg, or less (Section 12.3) can be mighty handy. But piping to a pager doesn't always work the way you want it to.

Here's a grep command line that searches several files. What's wrong with it?

% grep "^set" */.cshrc | more

That wasn't a fair question because you can't tell what's wrong. The problem (it turns out) is that the files named barney/.cshrc, edie/.cshrc, and gail/.cshrc are read-protected (Section 50.2). But as the first part of Figure 43-1 shows, the error messages scroll off your screen and the pager doesn't stop them.

Figure 43-1: Standard error bypassing pipe, going through pipe

Unless your display is reallllly sloooowww, the error messages are lost, and you never know they were there, or the errors are jumbled up with the "good" grep output. That's because you've told the shell to send only the standard output of grep to the pager program. And grep writes its errors to the standard error ( Section 36.15)! But both stdout and stderr go to the screen at once. The errors on stderr scroll away with the output from the pager. The pager can't count the lines of errors, so it outputs a complete screenful of stdout (the "good stuff"). If grep's standard output (from the files it could read) is at least a screenful, as it is here, there are too many lines to fit on the screen — and some lines will scroll off.

The better way to do this is to combine grep's stdout and stderr and give them both to the pager. These command lines (in csh and sh) both do that:

% grep "^set" */.cshrc |& more

$ grep "^set" */.cshrc 2>&1 | more

(The Z shell understands both.) The second part of Figure 43-1 shows how this works. Any time I pipe a command's output to a pager, I usually combine the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The > (right angle bracket) operator redirects the standard output of a process to a file. It doesn't affect the standard error. If you're logged in and can see any messages written to standard error, that's okay:

% nroff -ms report.ms > report.out &

[1] 10316

   ...Later...

nroff: can't open file /hoem/jpeek/report.data

But if you log out and leave the job running, you'll never see those errors unless you use the csh operator >& . It redirects both standard output and standard error to a file. For example:

make Section 11.10

% make >& make.output &

[1] 10329

% logout

               ...Later...

% cat make.output

        cc -O -c random.c

        cc -O -c output.c

"output.c", line 46: syntax error

"output.c", line 50: time_e undefined

"output.c", line 50: syntax error

   ...

You might also use the >& operator while you're logged in and watch the output file with tail -f ( Section 12.10). If you don't want the errors mixed with other output, you can split them to two files; see Section 43.1.

The C shell also has a pipe operator, |&, that redirects both standard output and standard error. It's great for running a job in the background or on another computer and mailing (Section 1.21) any output to me:

% make |& mailx -s "'make bigprog' output" jpeek@jpeek.com &

[1] 29182 29183

If I'd used plain | instead of |&, any text on the standard error wouldn't go into the mail message.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Have you ever destroyed a file accidentally? If you set the noclobber C shell variable or the noclobber option in bash, zsh, and ksh, it can help you avoid these mistakes. Setting noclobber prevents you from destroying a file when you are redirecting standard output (Section 43.1).

Consider the following situation:

% anycommand 

               > outputfile

The command above overwrites the old outputfile. If you have misspelled the name of your output file, or if you have forgotten that the file already exists and contains important data, or (most common) if you really meant to type >> instead of > (i.e., if you really meant to append to the end of outputfile, rather than start a new one), tough luck; your old data is gone.

Setting noclobber prevents this problem. If noclobber is set, the shell will not allow I/O redirection to destroy an existing file, unless you explicitly tell it to by adding an exclamation point (!) after the C shell redirect symbol or by adding a vertical bar (|) in ksh and bash. (The Z shell understands both.) Here are examples. The left column shows csh and tcsh; the right column is for bash (ksh is similar):

% set noclobber              $ set -o noclobber

% ls                         $ ls

filea fileb                  filea fileb

% anyprogram > fileb         $ anyprogram > fileb

fileb: File exists.          bash: fileb: Cannot clobber existing file

% anyprogram >! fileb        $ anyprogram >| fileb

%                            $

Be sure to put space after the !. If you don't, the C shell thinks you're making a history reference and it (usually) prints an error like fileb: Event not found.

Remember that noclobber is not an environment variable, so any new shells you create won't inherit it (Section 35.9). Therefore, if you want this feature, put the set command (above) in your shell's setup file (Section 3.3).

In some shells, noclobber will prevent you from redirecting standard output to /dev/null (Section 43.12) or to a terminal unless you add the !.

The noclobber variable has one other feature that's worth noting. Normally, shells let you append to a file that doesn't exist. If

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A useful shell trick is to use parentheses, ( ), to group commands.

The parentheses start a subshell (Section 24.4) that, in effect, "collects" the output of all the commands inside. (It does the same thing for the standard input and standard error.) The output of the entire group can be passed together into a single pipeline. For example:

echo Section 27.5

$ (cat file1; echo .bp; cat file2) | nroff

This will interpose the nroff .bp (break page) request between two files to be formatted.

Parentheses are also useful in the Bourne shell if you want to put an entire sequence of commands separated by semicolons (;) (Section 28.16) into the background. In the C shell, the command line below will go immediately into the background.

% nroff -ms file1; nroff -ms file2 &

But in the Bourne shell, the background request (&) will apply only to the second command, forcing you to wait for completion of the first job before you get back the system prompt. To get right back to work, you can type:

$ (nroff -ms file1; nroff -ms file2) &

Commands that run between the parentheses won't affect the parent shell's environment. For instance, to run a command in another directory without changing your active shell's current directory (Section 24.3):

% pwd

/home/trent

% (cd 

                  somewhere-else

                  ; nroff -ms file1 > file.out) &

[1] 22670

% pwd

/home/trent

The file file.out will be created in the somewhere-else directory.

—TOR and JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: tee

If you're running a program and you want to send its output to a file — but you want to see the output on your screen, too, so you can stop the program if something goes wrong — you can use tee. The tee program reads its standard input and writes it to one or more files. (The web site has the GNU version.)

A pipe may buffer the output of a program, collecting it in chunks and spitting it out every so often. If the program's output comes slowly and feeds tee through a pipe, there might be long delays before you see any output. In that case, it's better to use > to redirect output to a file, put the program into the background, and watch the output with tail -f (Section 12.10). Or use a program like script (Section 37.7).

Use tee for saving results in the middle of a long pipeline of commands. That's especially good for debugging. For example, you could type:

% prog 

               | tee prog.out | sed -f sedscr | tee sed.out | ...

to save the output of prog in the file prog.out and also pipe it to the sed command, save sed's output in sed.out and also pipe it, and so on.

Here are two other notes about tee. If you want to add to a file that already exists, use the -a option. tee can write to more than one file if you give all of the filenames as arguments

Z shell users usually don't need tee because they have the zsh MULTIOS option. For instance, here's how to write the pipeline above:

zsh% setopt multios

zsh% prog

                

               > prog.out | sed -f sedscr > sed.out | ...

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The tee (Section 43.8) command writes its standard input to a file and writes the same text to its standard output. You might want to collect several commands' output and tee them all to the same file, one after another. The obvious way to do that is with the -a option:

$ some-command 

               | tee teefile

$ another-command 

               | tee -a teefile

$ a-third--command 

               | tee -a teefile

A more efficient way is:

> Section 28.12

$ (

               some-command

> another-command

> a-third-command

               ) | tee teefile

The subshell operators (Section 43.7) collect the standard output of the three commands. The output all goes to one tee command. This has two differences from the first method. First, you need two fewer pipes, two fewer tees, and one more subshell. Second, you can pipe the output of the single tee command to another process — for example, to print it.

Unfortunately, the C shell doesn't make this quite as easy. If you can type all the commands on one line, you can do it this way (the same thing works in the Bourne shell):

% (

               command1; command2; command3

               ) | tee teefile

Otherwise, use a semicolon and backslash (;\) at the end of each line:

% (

               ;\



               ;\



               ) | tee teefile

In all these examples, remember that if you don't need to see the output of the commands, you don't need tee. Use the subshell as above, but replace | tee teefile with > outfile or | somecommand.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

What if you want to use the output of a program more than once, and you don't want to deal with an intermediary file? For example, suppose I have some large, compressed PostScript files. I want to print the files, but I also want to know how many pages they are. I know that the number of pages appears on a line following %%Pages: at the end of the file. Using bzcat (Section 15.6) to uncompress the file to standard output, I can type the following commands into a for loop (Section 28.9) (or put them into a shell script). This loop sends each file to the printer and uses sed to capture the correct line:

-n Section 34.3

for f

do

    bzcat $f | lpr

    bzcat $f | sed -n "s/^%%Pages: \([0-9][0-9]*\)/$f:   \1 pages/p"

done

But this ends up running bzcat twice, which takes some time. I can expand the file with bunzip2 first, but frankly I'm not sure I have the disk space for that.

Using process substitution and tee (Section 43.8), I can do it in one line, without wasting processes and without eating disk space:

for f

do

  bzcat $f | tee >(lpr) | sed -n "s/^%%Pages: \([0-9][0-9]*\)/$f: \1 pages/p"

done

From running this script, as each file is sent to the printer I receive the following messages on my screen:

ch01.ps.gz: 44 pages

ch02.ps.gz: 51 pages

ch03.ps.gz: 23 pages

   ...

Because tee can write to more than one file, it can write to more than one process with process substitution. For instance, maybe you want to send the file to both a black-and-white printer and a color printer at the same time:

bzcat $f | tee >(lpr -Pbw) >(lpr -Pcolor) | \

  sed -n "s/^%%Pages: \([0-9][0-9]*\)/$f: \1 pages/p"

Go to http://examples.oreilly.com/upt3 for more information on: tpipe

If your shell doesn't have process substitution, maybe you have a shell like bash or zsh that does. (Write a shell script. Or type the shell's name at your shell prompt, then type exit when you're done with the temporary shell.) Otherwise, you can use tpipe

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you type a pipe symbol (|) on a command line, the two processes that communicate through the pipe must both have been started from that same shell. Newer versions of Unix have a way to let two unrelated processes (processes not started from the same parent process) communicate: a named pipe or FIFO (First In First Out).

A FIFO works like a pipe, but its interface looks like a file. It has a filename and permissions (Section 1.17), and it's in a directory. Once you make the FIFO, one process can write to it (with the shell's > operator, or directly) and another process can read from it (the shell's < operator, or directly). Unlike a regular file, though, a FIFO doesn't "fill up" with data as a process writes to it: if there's no process waiting to read the data, the data is lost. So, when you use a FIFO between two processes, the processes still need to coordinate with each other. There are times that temporary files are better.

Go to http://examples.oreilly.com/upt3 for more information on: mkfifo

The command to make a FIFO is mkfifo . (The GNU version is on the CD-ROM [see http://examples.oreilly.com/upt3].) Like other files, the default permission is set by your umask. There's also a -m option that sets the permissions — with a numeric or symbolic mode like chmod (Section 50.5) uses. To remove a FIFO, use — you guessed it — rm.

Let's look at an example that, although it's made up, shows some important things to know about FIFOs. If you're using a window system, you'll use two terminal windows (like xterm (Section 24.20)); you'll write to the FIFO from one window and read it from the other. Or if you have two terminals, you can use both of them. Otherwise, with a single terminal, you can put the writing process in the background (Section 23.2) and run the reading process in the foreground.

Start by making the FIFO. You can make it from any window. (The FIFO stays in the filesystem until you remove it. You can use it over and over again, though only one pair of processes can use it at any one time.) Then have a look with

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It isn't a file, actually, though you can use it like one. /dev/null is a Unix device. It's not a physical device. /dev/null is a special device that "eats" any text written to it and returns "end-of-file" (a file of length 0) when you read from it. So what the heck can you use it for?

Empty another file. Just copy /dev/null "on top of" the other file (Section 15.2).
Make another program "quiet" by redirecting its output there. For instance, if you're putting a program into the background and you don't want it to bother you, type:

% progname 

               > /dev/null &

That redirects (Section 43.1) standard output but leaves standard error hooked to your terminal, in case there is an error.
Answer a program that asks a lot of questions — you know you'll just press RETURN at each prompt. In a lot of cases, you can redirect the program's standard input from /dev/null:
```
% progname 

                     < /dev/null

Want the default setup? If yes, press RETURN:

Enter filename or press RETURN for default:

   ...
```
You should test that with each program, though, before you assume this trick will work. (If it doesn't work, try yes (Section 14.5).)
Where a program needs an extra filename but you don't want it to read or write an actual file. For instance, the grep (Section 13.1) programs won't give the name of the file where they find a match unless there are at least two filenames on the command line. When you use a wildcard in a directory where maybe only one file will match, use /dev/null to be sure that grep will always see more than one (Section 9.21):

% grep "

               outputfile

               " * /dev/null

You're guaranteed that grep won't match its regular expression in /dev/null.
Section 15.3 shows even more uses for /dev/null.

Another interesting device (mostly for programmers) is /dev/zero. When you read it, you'll get ASCII zeros (NUL characters) forever. There are no newlines either. For both of those reasons, many Unix commands have trouble reading it. If you want to play, the command below will give you a start (and

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Your Unix machine can likely talk to a wide collection of hardware: disk controllers and disks (Section 44.4, Section 44.5), CD-ROMs (Section 44.6), ethernet cards (Section 44.8), modems (Section 44.10), sound cards (Section 44.13), and so on. Each device needs its own little piece of software within the kernel, called a device driver. Some device drivers are simple, and some are very complex; some cover multiple devices, and some are specific to one particular piece of hardware.

Many modern Unix platforms use loadable kernel modules for most device drivers, so that drivers can be loaded at run time rather than compiled into the kernel.

Many devices also have user-space tools to configure them, like ifconfig (Section 44.8) for network devices (Section 44.6, Section 44.7), mount (Section 44.9) for disks and so forth.

In this chapter we'll give you the whirlwind overview of devices on Unix. Since there are so many devices and so many platforms, we'll gloss over a lot of details, but hopefully this will give you enough to get started with and a few hints as to where to find more information.

— DJPH

As your Unix machine boots up, it will display a message for each device driver as it initializes. This is a good way to tell what devices your kernel was able to find. The exact output varies, but here is the output for hard drive controllers, hard drives, and network cards from a FreeBSD machine and a Debian Linux machine:

# FreeBSD

atapci0: <Intel ICH ATA66 controller> port 0xffa0-0xffaf at device 31.1 on pci0

ata0: at 0x1f0 irq 14 on atapci0

ata1: at 0x170 irq 15 on atapci0

ad0: 19569MB <ST320430A> [39761/16/63] at ata0- master UDMA66

afd0: 239MB <IOMEGA ZIP 250 ATAPI> [239/64/32] at ata0-slave using PIO3

acd0: CDROM <ATAPI CDROM> at ata1-master using PIO4

rl0: <D-Link DFE-530TX+ 10/100BaseTX> port 0xbc 00-0xbcff

    mem 0xefdfff00-0xefdfffff irq 11 at device 4.0 on pci1



# Linux

PIIX4: IDE controller on PCI bus 00 dev 39

PIIX4: not 100% native mode: will probe irqs later

    ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:pio

    ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:pio, hdd:pio

hda: WDC WD307AA-32BAA0, ATA DISK drive

ide0 at 0x1f0-0x1f7,0x3f6 on irq 14

hda: WDC WD307AA-32BAA0, 29333MB w/2048kB Cache, CHS=3739/255/63, UDMA

Partition check:

 hda: hda1 hda2 hda3

rtl8139.c:v1.07 5/6/99 Donald Becker

    http://cesdis.gsfc.nasa.gov/linux/drivers/rtl8139.html

eth0: RealTek RTL8139 Fast Ethernet at 0xd400, IRQ 11, 00:50:ba:d3:9e:14.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Your Unix machine can likely talk to a wide collection of hardware: disk controllers and disks (Section 44.4, Section 44.5), CD-ROMs (Section 44.6), ethernet cards (Section 44.8), modems (Section 44.10), sound cards (Section 44.13), and so on. Each device needs its own little piece of software within the kernel, called a device driver. Some device drivers are simple, and some are very complex; some cover multiple devices, and some are specific to one particular piece of hardware.

Many modern Unix platforms use loadable kernel modules for most device drivers, so that drivers can be loaded at run time rather than compiled into the kernel.

Many devices also have user-space tools to configure them, like ifconfig (Section 44.8) for network devices (Section 44.6, Section 44.7), mount (Section 44.9) for disks and so forth.

In this chapter we'll give you the whirlwind overview of devices on Unix. Since there are so many devices and so many platforms, we'll gloss over a lot of details, but hopefully this will give you enough to get started with and a few hints as to where to find more information.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As your Unix machine boots up, it will display a message for each device driver as it initializes. This is a good way to tell what devices your kernel was able to find. The exact output varies, but here is the output for hard drive controllers, hard drives, and network cards from a FreeBSD machine and a Debian Linux machine:

# FreeBSD

atapci0: <Intel ICH ATA66 controller> port 0xffa0-0xffaf at device 31.1 on pci0

ata0: at 0x1f0 irq 14 on atapci0

ata1: at 0x170 irq 15 on atapci0

ad0: 19569MB <ST320430A> [39761/16/63] at ata0- master UDMA66

afd0: 239MB <IOMEGA ZIP 250 ATAPI> [239/64/32] at ata0-slave using PIO3

acd0: CDROM <ATAPI CDROM> at ata1-master using PIO4

rl0: <D-Link DFE-530TX+ 10/100BaseTX> port 0xbc 00-0xbcff

    mem 0xefdfff00-0xefdfffff irq 11 at device 4.0 on pci1



# Linux

PIIX4: IDE controller on PCI bus 00 dev 39

PIIX4: not 100% native mode: will probe irqs later

    ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:pio

    ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:pio, hdd:pio

hda: WDC WD307AA-32BAA0, ATA DISK drive

ide0 at 0x1f0-0x1f7,0x3f6 on irq 14

hda: WDC WD307AA-32BAA0, 29333MB w/2048kB Cache, CHS=3739/255/63, UDMA

Partition check:

 hda: hda1 hda2 hda3

rtl8139.c:v1.07 5/6/99 Donald Becker

    http://cesdis.gsfc.nasa.gov/linux/drivers/rtl8139.html

eth0: RealTek RTL8139 Fast Ethernet at 0xd400, IRQ 11, 00:50:ba:d3:9e:14.

More specifically, in the line:

atapci0: <Intel ICH ATA66 controller> port 0xffa0-0xffaf at device 31.1 on pci0

atapci is the name of the device; 0 is the number of the device (devices are generally numbered sequentially with the first one probed getting the number 0); <Intel ICH ATA66 controller> is the name of the specific driver that successfully attached to this device; port 0xffa0-0xffaf at device 31.1 is physical address information about where this particular device is located; and finally, on pci0 tells us this device is attached to the first PCI bus (since pci is the device name of a PCI bus and 0 is the number assigned to the first PCI bus probed).

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Generally a Unix kernel is made up of some core, which handles fundamental functionality like virtual memory, and a lot of modules for various devices. A kernel configuration file is used to build a kernel and, on some platforms, a set of loadable kernel modules.

A kernel configuration file has a list of kernel options and then a list of devices and device options. The kernel build process uses this file to determine exactly what to build; this way you can have a kernel that supports exactly the hardware you have in your machine but isn't using any extra resources to support hardware you don't have.

Some example device lines from various kernel configuration files:

#

# FreeBSD samples

#

maxusers        128

options         INCLUDE_CONFIG_FILE

options         INET                    #InterNETworking

device          isa

device          pci

device          ata0    at isa? port IO_WD1 irq 14

device          ata

device          atadisk                 # ATA disk drives

device          atapicd                 # ATAPI CDROM drives

device          atapifd                 # ATAPI floppy drives

device          atapist                 # ATAPI tape drives

options         ATA_STATIC_ID           #Static device numbering



#

# Linux samples

#

# Loadable module support

CONFIG_MODULES=y

CONFIG_MODVERSIONS=y

# CONFIG_KMOD is not set



# General setup

CONFIG_NET=y

CONFIG_PCI=y



# Block devices

CONFIG_BLK_DEV_FD=m

CONFIG_BLK_DEV_IDE=y

# CONFIG_BLK_DEV_HD_IDE is not set

CONFIG_BLK_DEV_IDEDISK=y

CONFIG_BLK_DEV_IDECD=m

CONFIG_BLK_DEV_IDETAPE=m

CONFIG_BLK_DEV_IDEFLOPPY=m

# CONFIG_BLK_DEV_IDESCSI is not set

CONFIG_BLK_DEV_IDEPCI=y

CONFIG_BLK_DEV_IDEDMA=y

CONFIG_IDEDMA_AUTO=y

The kernel build process involves setting up an appropriate configuration file for your platform and then using a tool (generally config(8); check the manpage) to create a kernel build setup from the configuration file. Then you simply run make within the kernel build setup and you have a new kernel. Once the new kernel is installed, you reboot the machine, and poof, you're running on a sleek new customized kernel.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A physical disk can be divided into smaller blocks, called partitions. Unix disk devices operate on partitions, where each device is a single partition. The simplest configuration is one big partition for the entire disk.

The advantage to having filesystems on separate partitions is that different parts of your operating system are somewhat protected from each other. If your users have filled up /home, programs writing log files in /var aren't affected if /home and /var are separate partitions. If your disk gets corrupted, only the corrupted partition is damaged. The disadvantage is that, in most cases, if you mistakenly allocated too little disk space for a partition, you can't steal space from your /var to give you more room on /home once your system is set up.

On non-PC hardware, partitioning is generally simple enough; use format or disklabel to write a partition table onto the disk. Traditionally, partitions are named with a letter following the device name, for example, /dev/ad0a, /dev/ad0c and so forth. By convention, partition a is for a root filesystem (/), b is for swap space, c represents the whole disk, and so forth. Of course, every current platform changes this in some way. Check the manpages for the various tools mentioned for more details on what to do for your specific platform.

Solaris's disk device naming scheme is /dev/dsk/c?t?d?s?, where each ? is a number. The c is for controller, the t for target (a physical address on the controller), the d for disk, and the s for slice, another concept like partition. In this case, rather than partition c representing the whole disk, slice 2 does. This set of four numbers uniquely identifies a specific partition (slice) on a specific disk. Solaris uses format to manipulate partition tables.

On PC hardware, it's a bit more complicated, because the PC BIOS has a concept of partitions built into its understanding of disks. Unixes like Linux and FreeBSD that run on this hardware need to coexist with this partition table, especially if you want a machine that can dual-boot Unix and Windows. The BIOS understands no more than four

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

A filesystem is the scheme used to organize files on the disk. In the Windows world, FAT, FAT32, and NTFS are all filesystems. Various Unixes have their own filesystems with a forest of names: ufs, ext2fs, vxfs, ffs, nfs, mfs, ISO9660 (which most CD-ROMs use) and special filesystems like tmpfs, procfs, and devfs.

Filesystems like ufs (Unix File System), ffs (Fast File System), vxfs (Veritas Extended File System), and ext2fs (Extended File System, Version 2) are simply ways of organizing inodes and bytes with various strengths and weaknesses. nfs (Network File System) is a filesystem for making remote files appear to be available locally. mfs (Memory File System) is a filesystem for ramdisks, that is, file storage in memory instead of on disk. tmpfs (Temporary File System) is a file system often used for /tmp which shares filespace and swap space dynamically. procfs (Process File System) simulates a filesystem, but with process information in it instead of files. (procfs on Linux is different from procfs on the BSDs; FreeBSD has a linprocfs to simulate part of Linux's procfs.) devfs is similar, but for devices instead of processes.

Standard mounts are configured using /etc/fstab (or, on some platforms, /etc/vfstab). fstab is just a list of filesystems that should be mounted, along with where they should get mounted, what type of filesystem each device contains, and any options. My FreeBSD fstab looks like this:

# Device                Mountpoint      FStype  Options         Dump    Pass#

/dev/ad0s1b             none            swap    sw              0       0

/dev/ad2s1b             none            swap    sw              0       0

/dev/ad0s1a             /               ufs     rw              1       1

/dev/ad2s1e             /home           ufs     rw              2       2

/dev/ad0s1f             /usr            ufs     rw              2       2

/dev/ad0s1e             /var            ufs     rw              2       2

/dev/acd0c              /cdrom          cd9660  ro,noauto       0       0

proc                    /proc           procfs  rw              0       0

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Removable disks are prevalent in Unix machines; CD-ROMs, DVD-ROMs, Zip disks, and floppies are all removable disks. When a Unix system boots, normal filesystems are all mounted automatically. By definition, removable filesystems may not even be in the machine at boot time, and you certainly don't want to have to reboot your machine just to change CDs.

To do this, you use mount and umount . The -t option allows you to specify the type of filesystem. On my FreeBSD machine, I can mount a FAT-formatted Zip disk with:

# mount -t msdos /dev/afd0s4 /zip

If I've formatted the Zip disk with a BSD ufs filesystem instead, I don't need the -t option, since ufs is the default on FreeBSD, and I would use the BSD partitioning scheme (/dev/afd0c) instead of the BIOS partitions (/dev/afd0s4).

If you use your removable disk regularly, you can add it to your fstab and make this simpler:

/dev/acd0c              /cdrom          cd9660  ro,noauto       0       0

/dev/afd0c              /zip            ufs     rw,noauto       0       0

/dev/afd0s4             /mszip          msdos   rw,noauto       0       0

Note that I've set up my fstab for both ufs-formatted and FAT-formatted Zip disks, and that the Zip drive and the CD-ROM are both set noauto to keep them from being automatically mounted. Having these in my fstab means I can just type mount /zip or mount /cdrom to mount a Zip disk or CD-ROM. Don't forget to create the directories /cdrom, /zip, and /mszip!

Generally the mount and umount commands must be run as root. However, you'd often like normal users to be able to mount and unmount removable disks. Linux has an easy way to do this: just add user to the options field in /etc/fstab and normal users will be able to mount and unmount that device. (Incidentally, Linux also has an auto filesystem type, which is very handy for removable devices, because it does its best to dynamically figure out what filesystem is on the removable media.) On other platforms, it can be a little more complex. Generally, the trick is to set the permissions on the device file properly. On FreeBSD you also need to use

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Some platforms provide the capability to mount a file as if it were a block device (like a disk partition (Section 44.4)). This allows mounting a file as if it were a hard disk, CD-ROM, or any other physical media. The primary advantage to this is that it's a simple way to create or work with a floppy, Zip, or CD-ROM image without needing the physical device. You can mount a CD image without having to burn an actual CD or manipulate a floppy boot image. Of course, different platforms call it different things and use different tools.

Mounting file images on Linux uses the loop device and is called a loop mount or a loopback mount. To mount an existing image as a filesystem, use the loop option to mount:

% mount -t iso9660 -o loop image.iso /mnt

% ls /mnt

To create a new image, you first create an empty file of the correct size (this is effectively creating a partition (Section 44.4) — in this case, a 100 megabyte image. You then attach the image to one of the available loop device and use mkfs to create a new filesystem in the image. Then you can mount the image normally. In this example, we'll release the loop device we had to allocate specifically and let the mount find an available loop device automatically.

% dd if=/dev/zero of=

               image.file 

               bs=1k count=100000

% losetup /dev/loop 

               image.file

% mkfs -c /dev/loop 

               100000

% losetup -d /dev/loop

% mount -o loop 

               image.file /mnt

FreeBSD has a similar capability, called vnode disks, with very similar syntax, but you use /dev/vn instead of /dev/loop and vnconfig instead of losetup. See FreeBSD's vnconfig(8) manpage.

Solaris also has loop devices as of Solaris 8. The device is /dev/lofi instead of /dev/loop, and you use lofiadm to configure it. See Solaris's lofiadm(1M) and lofi(7D) manpages.

FreeBSD and Solaris don't provide an equivalent to the loop option to mount; instead you just use vnconfig or lofiadm to explicitly associate a particular block device with the file and mount the specific block device just like any other device.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

ifconfig is used to configure network devices such as Ethernet cards. While booting, the kernel will find a device driver for the actual device, but it will still need to be assigned an IP address, and any protocol options need to be configured. Various platforms have different ways to store this configuration information, but most use ifconfig somewhere in the startup scripts to do the actual work.

The primary use of ifconfig is to set up a network device to use a particular IP address. ifconfig can also be used to set network options and aliases. To bring up an interface (in this case, rl0) on 192.168.1.1 with normal settings for a /24 network:

# ifconfig rl0 inet 192.168.1.1 netmask 255.255.255.0 broadcast 192.168.1.255 up

To temporarily bring a network interface down and then back up later, something that can be useful for maintenance:

# ifconfig rl0 down

# ...maintenance operations...

# ifconfig rl0 up

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Network filesystems provide the illusion that files on a remote host are on your disk. Except for mounting and unmounting such a filesystem and but for a few low-level details, they can be treated like any local filesystem, albeit on a very slow disk. The two most common network filesystems available on Unix platforms are the Network File System (NFS) and Server Message Block File System (SMBFS).

NFS has been around for a long time and is available on every Unix system I've seen in the past ten years. Its interface is simple: an NFS server has a set of exported filesystems (usually listed in /etc/exports), and any permitted client can mount those filesystems using a straightforward mount invocation. Simply specify host:/filesystem as the device, and tell mount that the filesystem is of type nfs:

# mount -t nfs orange:/home /orange

For more details on NFS on your platform, take a look at the manpages for exports(5) and mount_nfs(8) or nfs(5).

NFS mounts can hang up entirely if the NFS server goes down or if you lose your net connection to it. Often this can require rebooting your machine to fix. To avoid this, use the soft option when mounting NFS filesystems. soft tells the NFS client system to use timeouts, so that losing touch with the NFS server just causes I/O requests to time out instead of hanging your machine.

NFS by itself is extremely insecure. Be aware that running NFS without any other precautions on a publicly accessible network opens you up to a wide variety of attacks. http://nfs.sourceforge.net/nfs-howto/security.html addresses some of the issues involved and has links to other good information on the subject.

SMB is the primary file and printer sharing protocol used by Windows. Chapter 47 details Samba, the primary tool used to deal with SMB on Unix systems. smbfs is the tool used to mount SMB-shared filesystems (including Windows shared drives and the like) as if they were Unix filesystems. Much like NFS,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The word "modem" is a contraction of "modulator-demodulator." The fundamental job of a modem is to turn a digital signal into an analog signal and send that analog signal across a phone line (modulation) and to receive an analog signal from a phone line and turn it back into the original digital signal (demodulation).

Controller-based modems do all of the digital signal processing, D/A and A/D conversion, and phone-line interfacing in hardware. Generally, these modems either are external modems that plug into a serial port or have a serial port chip included and thus just look like an extra serial port to the CPU. Configuring these modems under Unix is easy; just set up whatever program uses the serial port to use the port speed and serial options you want.

Host-based modems, often called "Winmodems," provide some level of hardware support (at a minimum, the physical phone line interface) and then emulate some or all of the hardware modulation and demodulation in software. There are a variety of specifications related to "soft" modems, and current information on things like available drivers, issues, standards, and whether a modem is a hard or soft modem are available at http://www.idir.net/~gromitkc/winmodem.html and http://www.linmodems.org.

The problem that soft modems present to Unix is that the software that makes up the fundamental functionality of the modem is almost always Windows software. These modems are widely available and cheap and do have some advantages, though, so there are efforts to provide Unix software for some set of them. Unix soft-modem software is highly in flux at the time of this writing. Before you buy a modem, be sure that you check the current information on that modem and available drivers for the Unix platform you want to use before you buy. Or spend a bit more and buy a modem that doesn't have these issues.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Point-to-Point Protocol (PPP) is the way ISPs usually provide dialup access (largely because this is the default protocol Windows dialup uses). Unixes that can do dialup provide a PPP client, which you configure to call the ISP and set up a PPP connection. An established connection functions as a network connection — you can use ifconfig (Section 44.8, Section 46.3) to examine it and packets will be routed to the PPP connection by default, and tools like traceroute (Section 46.4) can be used across it.

Unixes provide two ways to run PPP: kernel PPP, where the PPP code resides in the kernel and is therefore very fast but limited in features, and user PPP, where packets have to be copied back and forth between kernel space and user space, but a wide feature set can be provided. We'll give a quick overview of both.

Kernel PPP uses pppd and a fairly simple set of configuration commands. You provide pppd with the information needed to dial your modem appropriately and with whatever login information your ISP has provided you, and it connects. Generally you then have to set up /etc/resolv.conf to point to your ISP's DNS (Section 46.9) server. Some implementations of pppd don't even know how to dial the phone, and you'll have to use something like kermit to dial the phone first. pppd must also be run as root. Look at your platform's documentation for pppd for details on setting up kernel PPP on that platform.

Platforms that provide a user-space PPP client are a little easier to work with. User-space PPP clients can be run by users other than root (usually limited to a specific group); they tend to configure default routes, /etc/resolv.conf, and other details automatically; and they generally deal with PAP or CHAP authentication (which many ISPs use) a little more easily. Usually the user-space PPP client is just called ppp; look for its manpage to see what it requires to configure it.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Many PCs support the Universal Serial Bus (USB). USB is a hot-swappable standard; devices can be plugged in and unplugged while the machine is running, and the system is supposed to recognize the new device or no longer recognize the now disconnected device.

Unixes deal with this requirement with low-level device drivers to actually interface with the devices and with a daemon, usbd, to monitor for changes on the fly or, on Linux, the hotplug facility (http://linux-hotplug.sourceforge.net).

Generally, there is very little configuration required for supported USB devices. If you have the correct kernel modules (Section 44.3) loaded (and on many platforms they're loaded by default), just plug in the device. Check your platform's supported hardware before buying a USB device, as such devices are changing rapidly at the time of this writing and may or may not have Unix drivers implemented yet.

Specific issues you might run into include that USB disks may need to use a special filesystem type (usbdevfs) and that specific devices may require tools to actually use the device. Webcams and scanners are a good example, as the device driver provides only low-level access to the device; you still need a tool that can pull images off of the device and do something useful with them. Extensive information is available on the Web about using many USB devices on the free Unixes (http://www.linux-usb.org for Linux and the USB chapter in the FreeBSD handbook are places to start), and it stays fairly up to date.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are a lot of devices available for PCs that were never designed for an operating system like Unix to use. Often these devices' manufacturers simply provide Windows drivers and never expect you to need anything else. Luckily, there is a large community of developers for the various free Unixes, and they implement device drivers for many of these devices. Availability of a driver for a particular piece of hardware, however, depends entirely on whether someone happened to write a driver for it.

Sound cards are one bit of hardware that commonly has this problem. Most free Unixes have a set of drivers that support a selection of sound cards and one or two other drivers that support a lowest common denominator to get minimal functionality out of most sound cards. If you want real support for your sound card, look at the supported devices list for the OS you want to install before you buy a card, and pick one that someone's written a full driver for.

On Linux, take a look at the sndconfig utility, which can probably configure your sound card for you. Take a peek at http://www.linuxheadquarters.com/howto/basic/sndconfig.shtml for details.

Other hardware that falls into the "check your supported hardware list before buying" includes frame grabbers, multi-serial boards, AD/DA converters, X-10 controllers and any hardware that's brand new (and thus may not have had time for someone to create a Unix driver). All of the free Unixes have extensive supported hardware lists — check before you buy.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Often server machines are placed in a rack in a colocation facility, in some back closet, or in some other out of the way place. This can make it really inconvenient to access the server's console should something go wrong or need diagnosing; hauling a monitor and keyboard into your server storage area is a real pain. If you've got your server mounted in a rack, there are devices that are essentially a flat screen monitor, keyboard, and mouse mounted in a sliding rack shelf, which work well, but they're expensive.

A simple and cheap solution is to change the console from the normal monitor/keyboard/mouse to one of the serial ports. The serial port can be hooked via null modem to a terminal server or another machine, allowing controlled access, or you can just plug your laptop into it with a null modem when you need to diagnose problems or reboot.

Linux has a howto describing details of dealing with serial consoles at http://www.linuxdoc.org/HOWTO/Remote-Serial-Console-HOWTO/. Essentially, you provide options to the boot loader and kernel to tell them to use your serial port as a console, and then configure getty to accept logins on that serial port. The HOWTO shows various potential configurations and demonstrates proper setup on each.

FreeBSD's handbook has a chapter on setting up serial consoles. Again, you have to tell the boot loader and the kernel to use the serial port, and then edit /etc/ttys to enable getty on that serial port. FreeBSD can also be configured to decide whether to use a normal console or serial console based on whether or not a keyboard is plugged in. NetBSD and OpenBSD are configured similarly.

Solaris is even easier: just unplug the keyboard before you boot the machine. Solaris uses a serial console by default if no keyboard is plugged in at boot time. If you want to set it explicitly to use a serial console even if the keyboard is plugged in, just set input-device and output-device to ttya (or ttyb if you want it on the second serial port) in the boot eeprom.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This chapter discusses printing, which is a surprisingly complicated subject. To understand why printing is so complicated, though, let's think a little bit about what you might want to print.

First, in the "olden days," we had line printers and their relatives: daisy-wheel printers, dot-matrix printers, and other pieces of equipment that generated typewriter-like output. Printing a simple text file was easy: you didn't need any special processing; you only needed some software to shove the file into the printer. If you wanted, you might add a banner page and do a little simple formatting, but that was really pretty trivial.

The one area of complexity in the printing system was the " spooling system," which had to do several things in addition to force-feeding the printer. Most printers were (and still are) shared devices. This means that many people can send jobs to the printer at the same time. There may also be several printers on which your file gets printed; you may care which one is used, or you may not. The spooling system needs to manage all this: receiving data from users, figuring out whether or not an appropriate printer is in use, and sending the file to the printer (if it's free) or storing the file somewhere (if the printer isn't free).

Historical note: why is this called the "spooling system"? Dave Birnbaum, a Principal Scientist at Xerox, says:

"SPOOL (Simultaneous Printing Off and On Line)" It was written for the early IBM mainframes (of the 3-digit, i.e., 709 kind) and extended to the early 1401 machines. Output for the printer was sent to the spool system, which either printed it directly or queued it (on tape) for later printing (hence the on/off line). There was also a 2nd generation version where the 1401 would act as the printer controller for the (by then) 7094. The two were usually connected by a switchable tape drive that could be driven by either machine." [There's some controversy about exactly what the acronym means, but Dave's is as good as any I've heard. — JP]

The first few articles in this chapter, Section 45.2, Section 45.3, Section 45.4, and Section 45.5, discuss the basic Unix spooling system and how to work with it as a user.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This chapter discusses printing, which is a surprisingly complicated subject. To understand why printing is so complicated, though, let's think a little bit about what you might want to print.

First, in the "olden days," we had line printers and their relatives: daisy-wheel printers, dot-matrix printers, and other pieces of equipment that generated typewriter-like output. Printing a simple text file was easy: you didn't need any special processing; you only needed some software to shove the file into the printer. If you wanted, you might add a banner page and do a little simple formatting, but that was really pretty trivial.

The one area of complexity in the printing system was the " spooling system," which had to do several things in addition to force-feeding the printer. Most printers were (and still are) shared devices. This means that many people can send jobs to the printer at the same time. There may also be several printers on which your file gets printed; you may care which one is used, or you may not. The spooling system needs to manage all this: receiving data from users, figuring out whether or not an appropriate printer is in use, and sending the file to the printer (if it's free) or storing the file somewhere (if the printer isn't free).

Historical note: why is this called the "spooling system"? Dave Birnbaum, a Principal Scientist at Xerox, says:

"SPOOL (Simultaneous Printing Off and On Line)" It was written for the early IBM mainframes (of the 3-digit, i.e., 709 kind) and extended to the early 1401 machines. Output for the printer was sent to the spool system, which either printed it directly or queued it (on tape) for later printing (hence the on/off line). There was also a 2nd generation version where the 1401 would act as the printer controller for the (by then) 7094. The two were usually connected by a switchable tape drive that could be driven by either machine." [There's some controversy about exactly what the acronym means, but Dave's is as good as any I've heard. — JP]

The first few articles in this chapter, Section 45.2, Section 45.3, Section 45.4, and Section 45.5, discuss the basic Unix spooling system and how to work with it as a user.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix used a print spooler to allow many users to share a single printer long before Windows came along. A user can make a printing request at any time, even if the printer is currently busy. Requests are queued and processed in order as the printer becomes available.

Unix permits multiple printers to be connected to the same system. If there is more than one printer, one printer is set up as the default printer, and print jobs are sent there if no printer is specified.

Many systems use the lpr command to queue a print job. When you use lpr, it spools the file for printing.

$ lpr notes

The lpq command tells you the status of your print jobs by showing you the print queue for a given printer.

$ lpq

lp is ready and printing

Rank   Owner      Job  Files                Total Size

active fred       876  notes                7122 bytes

1st    alice      877  standard input       28372 bytes

2nd    john       878  afile bfile ...      985733 bytes

The word active in the Rank column shows the job that's currently printing. If your job does not appear at all on the listing, it means your job has finished printing or has been completely written into the printer's input buffer (or perhaps that you accidentally printed it to a different queue). If a job is not active, it's still in the queue.

You can remove a job with the lprm command. (Run lpq first to get the job number.)

$ lprm 877

dfA877host dequeued

cfA877host dequeued

The command lpc status (Section 45.3) can be used to determine which printers are connected to your system and their names. If there is more than one printer, you can then use the -P option with lpr, lpq and lprm to specify a printer destination other than the default. For instance, if a laser printer is configured as laserp, you can enter:

$ lpr -Plaserp myfile

If you'll be using a certain printer often, put its name in the PRINTER environment variable (Section 45.4).

If you're using an older system that has only lp (see below), or if you'd like a fancier

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The lpc (8) command, for lpr-style printing setups, is mostly for the superuser. (You may find it in a system directory, like /usr/sbin/lpc.) Everyone can use a few of its commands; this article covers those.

You can type lpc commands at the lpc> prompt; when you're done, type exit (or CTRL-d):

% lpc

lpc> help status

status          show status of daemon and queue

lpc> ...

lpc> exit

%

Or you can type a single lpc command from the shell prompt:

% lpc status imagen

imagen:

        queuing is enabled

        printing is enabled

        no entries

        no daemon present

%

The printer daemon (Section 1.10) watches the queue for jobs that people submit with lpr (Section 45.2). If queueing is disabled (usually by the system administrator), lpr won't accept new jobs.

lpc controls only printers on your local host. lpc won't control printers connected to other hosts, though you can check the queue of jobs (if any) waiting on your local computer for the remote printer.

The commands anyone can use are:

restart [printer]: This tries to start a new printer daemon. Do this if something makes the daemon die while there are still jobs in the queue (lpq or lpc status will tell you this). It's worth trying when the system administrator is gone and the printer doesn't seem to be working. The printer name can be all to restart all printers. The printer name doesn't need an extra P. For example, to specify the foobar printer to lpr, you'd type lpr -Pfoobar. With lpc, use a command like restart foobar.
status [printer]: Shows the status of daemons and queues on the local computer (see the preceding example). The printer name can be all to show all printers.
help [command]: By default, gives a list of lpc commands, including ones for the superuser only. Give it a command name and it explains that command.
exit: Quits from lpc.

— JP

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Each printer on your system should have a name. By default, commands that send a file to a printer assume that the printer is named lp (a historical artifact; it stands for "Line Printer"). If you're using a single-user workstation and have a printer connected directly to your workstation, you can name your printer lp and forget about it.

In many environments, there are more options available: e.g., there are several printers in different locations that you can choose from. Often, only one printer will be able to print your normal documents: you may need to send your print jobs to a PostScript printer, not the line printer that the accounting department uses for billing.

There are two ways to choose a printer:

Printing commands in the lpr family accept the option -P printer. This includes lpr (Section 45.2), various scripts to format typeset documents, etc. For example, lpr -Pps file.ps sends the file file.ps to the printer named ps.
Commands in the lpr family recognize the PRINTER environment variable (Section 35.3); if PRINTER is defined, the command will read its value and choose a printer accordingly. So the command:
```
% setenv PRINTER ps                

                      — or

$ PRINTER=ps ; export PRINTER

                  
```
ensures that the lpr-style print commands will send your documents to the printer named ps. The -P option overrides this environment variable, in case you need to send a particular print job to a different printer.
Commands in the lp family use the -d option to select a printer. So lp -d prfile.ps sends file.ps to the printer named pr; it's equivalent to the previous lpr example.
Commands in the lp family look for an environment variable named LPDEST, rather than PRINTER. So:
```
% setenv LPDEST ps                

                      — or

$ LPDEST=ps ; export LPDEST

                  
```
ensures that the lp-style print commands will send your documents to the printer named ps. The -d option overrides this environment variable.

Note that Solaris and others that use lp can include both the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you print a file, the file is copied to a "spooling directory." This can be a problem if you want to print a very large file: the copy operation might take a long time, or the act of copying might fill the spooling directory's filesystem.

Systems with the lpr family of commands provide a workaround for this problem. The -s option makes a symbolic link (Section 10.4) to your file from the spooling directory.

Here's such a command:

% lpr -s directions

Rather than copying directions, lpr creates a symbolic link to directions. The symbolic link is much faster, and you're unlikely to get a "filesystem full" error.

Using a symbolic link has one important side effect. Because the file isn't hidden away in a special spooling directory, you can delete or modify it after you give the lpr command and before the printer is finished with it. This can have interesting side effects; be careful not to do it.

Of course, this warning applies only to the file that actually goes to the printer. For example, when you format a troff file (Section 45.16) for a PostScript printer and then print using -s, you can continue to modify the troff file, because it's the resulting PostScript file that actually goes to the printer (thus the PostScript file, not the troff file, is symbolically linked).

— ML

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The line printer spooler (Section 45.2) prints what you send it. If you send it a continuous stream of text (and the printer is set up to print text files rather than PostScript), that's probably just what you'll get: no page breaks, indenting, or other formatting features.

That's where pr comes in. It's a simple formatter that breaks its input into "pages" that will fit onto a 66-line page. (You can change that length.) It adds a header that automatically includes the date and time, the filename, and a page number. It also adds a footer that ensures that text doesn't run off the bottom of the page.

This is just what you want if you are sending program source code or other streams of unbroken text to a printer. For that matter, pr is often very handy for sending text to your screen. In addition to its default behavior, it has quite a few useful options. Here are a few common options:

-f: Separate pages using formfeed character (^L) instead of a series of blank lines. (This is handy if your pages "creep" down because the printer folds some single lines onto two or three printed lines.)
-h str: Replace default header with string str. See Section 21.15.
-l n: Set page length to n (default is 66).
-m: Merge files, printing one in each column (can't be used with -num and -a). Text is chopped to fit. See Section 21.15. This is a poor man's paste (Section 21.18).
-s c: Separate columns with c (default is a tab).
-t: Omit the page header and trailing blank lines.
-w num: Set line width for output made into columns to num (default is 72).
+ num: Begin printing at page num (default is 1).
-n

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

enscript is a handy program that takes your text files and turns them into PostScript. enscript comes with a wide variety of formatting options. There is a GNU version available, and a few Unixes include a version by default. enscript is particularly useful when your main printer speaks primarily PostScript.

Detailed information on everything enscript can do is available in its manpage, but here are a few examples:

% enscript -G stuff.txt

               Fancy ("Gaudy") headers

% enscript -2r stuff.txt

               Two-up printing -- two pages side-by-side on each page of paper

% enscript -2Gr stuff.txt

               Two-up with fancy headers

% enscript -P otherps stuff.txt

               Print to the 

               otherps

                printer instead of the default

% enscript -d otherps stuff.txt

               Ditto

% enscript -i 4 stuff.txt

               Indent every line four spaces

% enscript --pretty-print=cpp Object.cc

               Pretty print C++ source code

% enscript -E doit.pl

               Pretty print doit.pl (and automagically figure out that it's Perl from the .pl suffix)

One thing to watch for: enscript's default page size is A4, and in the United States most printers want letter-sized pages. You can set the default page size to letter when installing enscript (many U.S. pre-built binary packages do this for you), or you can use the -M letter or - -media=letter option when you call enscript.

If you want a default set of flags to be passed to enscript, set the ENSCRIPT environment variable. Anything you pass on the command line will override values in ENSCRIPT.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sometimes you'd like to be able to print to a printer that's physically attached to another Unix machine. lpd , the print spool daemon, supports this easily.

lpd is configured using the printcap printer capabilities database, generally stored in /etc/printcap. Generally, a local printer is given a line that looks something like this:

lp|local line printer:\

       :lp=/dev/lpt0:\

       :sd=/var/spool/output/lpd:lf=/var/log/lpd-errs:

The first line sets the printer name, in this case lp, and gives it a more descriptive name (local line printer) as well. The rest of the lines define various parameters for this printer using a parameter=value format. lp specifies the printer device — in this case, /dev/lpt0. sd specifies the local spool directory, that is, where lpd will store spooled files while it's working with them. lf specifies the log file, where lpd will write error messages and the like for this printer.

To set up a remote printer, all you have to do is provide a remote machine (rm) and a remote printer (rp) instead of a printer device:

rlp|printhost|remote line printer:\

       :rm=printhost.domain.com:rp=lp:\

       :sd=/var/spool/output/printhost:lf=/var/log/lpd-errs:

Note that we added another name; since this is the default printer for the host printhost, either rlp or printhost will work as printer names. We also used a different spool directory, to keep files spooled for printhost separate from local files; this isn't strictly necessary, but it's handy. Don't forget to create this spool directory before trying to spool anything to this printer!

Some network connected printers have lpd-compatible spoolers built in. Talking to one of these printers is just as easy; just provide the printer's hostname for rm. Generally you won't have to provide rp unless the printer supports different printing modes by using different remote printer names, since the default name lp is almost always supported by these sorts of printers.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Samba provides SMB networking to Unix boxes; in English, that means it allows Unix machines to share disks and printers with Windows machines and vice versa. Chapter 49 details Samba; here we'll talk a bit about tricks for printing over Samba, since it's so useful and parts of it are fairly arcane.

This is the easy one. Simply configure your printer normally using printcap, then set this in your smb.conf:

    load printers = yes

This tells Samba to read the printcap file and allow printing to any printer defined there. The default [printers] section automatically advertises all printers found and allows anyone with a valid login to print to them. You may want to make them browsable or printable by guest if you're not particularly worried about security on your network. Some Windows configurations will need guest access to browse, since they use a guest login to browse rather than your normal one; if you can't browse your Samba printers from your Windows client, try setting up guest access and see if that fixes it.

If you want to get really fancy, current versions of Samba can support downloading printer drivers to clients, just like Windows printer servers do. Take a look at the PRINTER_DRIVER2.txt file in the Samba distribution for more about how to do this.

This one's a little more tricky. lpd doesn't know how to print to a Windows printer directly, or how to talk to Samba. However, lpd does know how to run files through a filter (Section 45.17). So what we'll do is provide a filter that hands the file to Samba, and then send the print job right to /dev/null:

laserjet:remote SMB laserjet via Samba\

    :lp=/dev/null:\

    :sd=/var/spool/lpd/laser:\

    :if=/usr/local/samba/bin/smbprint:

Samba comes with a sample filter called smbprint ; it's often installed in an examples directory and will need to be moved to somewhere useful before setting this up.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Once upon a time, printers were simple. You hooked them up to your machine and dumped text out to them, and they printed the text. Nothing fancy, and not very pretty either. As printers got smarter, they became capable of more things, printing in a different font, perhaps. Printing got a bit more complex. If you wanted to use fancy features, you had to embed special characters in your text, specific to the printer.

Printers got even smarter, and could draw pictures, print images, and use all sorts of fonts. They started using complex languages (Section 45.14) to print, which made dealing with them more complex but at least somewhat more consistent. People wrote tools to convert text (Section 45.7) so it could be printed.

Webster defines typesetting as "the process of setting material in type or into a form to be used in printing," literally, the setting of type into a printing press. As computers have gotten more sophisticated, it has come to include the formatting of text and images to send to typesetting machines and then, later, smart printers. These days, your average printer is pretty smart and can handle everything the typesetters of old could do and more. Windows systems provide What You See Is What You Get (WYSIWYG, pronounced whiz-ee-wig) editors as a matter of course, most of which do all of their typesetting without any user intervention (and often badly, to boot).

On Unix, typesetting generally involves describing the formatting you want using a formatting language and then processing the source file to generate something that a printer can understand. There are a variety of tools and languages that do this, with various purposes, strengths, and weaknesses. Many formatting languages are markup languages, that is, they introduce formatting information by "marking up" the text you want formatted.

There is an entire science (and art) of typography that we won't try to get into here. My personal favorite books on the subject are Robert Bringhurst's The Elements of Typographic Style for general typography and Donald Knuth's

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix was one of the first operating systems to provide the capability to drive a typesetter. troff is both a markup language and a tool for generating typesetter output.

Originally, troff was designed to drive a device called a C/A/T phototypesetter, and thus it generated a truly frightening collection of idiosyncratic commands. For a while, there were several version of troff and troff-related tools, including tools to translate C/A/T output into something useful, versions of troff that output slightly saner things than C/A/T, and so forth. It was all very confusing.

Most systems these days still have a version of troff, often GNU's groff, which outputs PostScript and other device-independent formats. Unix manpages are still written in nroff, a related tool that takes the same input and spits out ASCII-formatted text, using the man macro package. However, most people don't use troff and its related tools for general text formatting much any more.

So why do we care about troff? The Jargon Dictionary (Version 4.2.2) has this to say:

troff /T'rof/ or /trof/ n.
The gray eminence of Unix text processing; a formatting and phototypesetting program, written originally in PDP-11 assembler and then in barely-structured early C by the late Joseph Ossanna, modeled after the earlier ROFF which was in turn modeled after the Multics and CTSS program RUNOFF by Jerome Saltzer (that name came from the expression "to run off a copy"). A companion program, nroff, formats output for terminals and line printers.
In 1979, Brian Kernighan modified troff so that it could drive phototypesetters other than the Graphic Systems CAT. His paper describing that work ("A Typesetter-independent troff," AT&T CSTR #97) explains troff's durability. After discussing the program's "obvious deficiencies — a rebarbative input syntax, mysterious and undocumented properties in some areas, and a voracious appetite for computer resources" and noting the ugliness and extreme hairiness of the code and internals, Kernighan concludes:

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The definitive documentation system for every Unix is manpages. (Much GNU software is documented fully in info pages instead, but manpages are so foundational that even those packages generally provide some sort of manpage.) What is a manpage, then?

A manpage is a text file, marked up with nroff commands, specifically using the man macro package. (Well, technically, using the tmac.an standard macro package — t/nroff takes a -m option to specify which tmac.* macro package to use. Thus, man uses nroff -man.) A simple manpage (in this case, the yes(1) manpage from FreeBSD) looks something like this:

.Dd June 6, 1993

.Dt YES 1

.Os BSD 4

.Sh NAME

.Nm yes

.Nd be repetitively affirmative

.Sh SYNOPSIS

.Nm

.Op Ar expletive

.Sh DESCRIPTION

.Nm Yes

outputs

.Ar expletive ,

or, by default,

.Dq y ,

forever.

.Sh HISTORY

The

.Nm

command appeared in

.At 32v .

This collection of difficult-to-read nroff commands, when formatted by nroff via the man command on my FreeBSD machine, looks something like this:

YES(1)                  FreeBSD General Commands Manual                 YES(1)



NAME

     yes - be repetitively affirmative



SYNOPSIS

     yes [expletive]



DESCRIPTION

     Yes outputs expletive, or, by default, "y", forever.



HISTORY

     The yes command appeared in Version 32V AT&T UNIX.



4th Berkeley Distribution        June 6, 1993                                1

The various nroff/man macros allow you to define things such as the name of the command, the short description of what it does, the list of arguments, and so forth, and formats it all into the standard look of a manpage. To write your own manpages, take a look at existing manpages for examples, and read the man(1) and man(7) manpages.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 45.12 shows an example of a simple formatting markup language; the one used by man via nroff. Don't laugh — it may seem arcane, but it is fairly simple. Like all markup languages, it attempts to abstract out certain things, to allow you to describe what you'd like the end result to look like. Manpages are simple to describe, so the markup language for them is relatively simple.

Full troff is somewhat more complex, both because it allows expressing far more complex ideas, and because it allows definition of macros to extend the core markup language. Similarly, T_EX (pronounced "tek") is essentially a programming language for typesetting. It provides a very thorough model of typesetting and the ability to, essentially, write programs that generate the output you want.

Available on top of T_EX is L^AT_EX (pronounced "lah-tek" or "lay-tek"), a complex macro package focused on general document writing. It allows you to describe the general structure of your document and let L^AT_EX (and underneath, T_EX) sort out the "proper" way to typeset that structure. This sort of markup is very different to deal with than working in a WYSIWYG word processor, where you have to do all of the formatting yourself. As an example, a simple L^AT_EX document looks something like this (taken from The Not So Short Introduction to LATEX2e):

\documentclass[a4paper,11pt]{article}

% define the title

\author{H.~Partl}

\title{Minimalism}

\begin{document}

% generates the title

\maketitle

% insert the table of contents

\tableofcontents

\section{Start}

Well, and here begins my lovely article.

\section{End}

\ldots{} and here it ends.

\end{document}

Much like the nroff input earlier, this describes the structure of the document by inserting commands into the text at appropriate places. The LyX editor (http://www.lyx.org) provides what they call What You See Is What You Mean (WYSIWYM, or whiz-ee-whim) editing by sitting on top of L^AT_EX. Lots of information about T_EX and L^AT_EX is available at the T_EX Users' Group web site,

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Printing languages, also sometimes called page description languages, are representations of exactly what needs to be on the screen or printed page. They are generally a collection of drawing commands that programs can generate, often with extra features to make drawing complex pictures or doing fancy things with text easier.

PostScript was developed by Adobe in the early 1980s to provide some sort of generic page description language. It is a fairly complete language; I've written complex PostScript programs by hand. This makes it much easier to write software that can generate PostScript output. Modern troffs can generate PostScript, and ghostscript can be used to process PostScript into printer-specific output for certain non-PostScript printers, so PostScript is a very useful output form.

Printer Command Language, or PCL, was originally developed by Hewlett-Packard, also in the early 1980s, to provide a generic printer language for their entire range of printers. Early versions were very simple, but PCL 3 was sophisticated enough that other printer manufacturers started to emulate it, and it became a de facto standard. PCL's more current incarnations are quite flexible and capable. Incidentally, ghostscript can turn PostScript into PCL, and most printers that can't speak PostScript can speak some form of PCL these days. My primary printer these days speaks PCL 5E, and I use it from both Windows machines and Unix machines.

DVI stands for "device independent" and is the primary output from T_EX (and thus L^AT_EX). Like PostScript, it's a generic language for describing the printed page. There are converters that convert DVI into PostScript, PCL and PDF.

PDF is Adobe's successor to PostScript. PDF has a special place on the web, because it's been promoted as a way to distribute documents on the web and have them displayed consistently in a wide variety of environments, something not possible in HTML. This consistency is possible for the same reasons any page description language can provide it: the focus of such a language is on describing exactly what the page should look like rather than being human readable or editable, like most markup languages. However, Adobe has provided Acrobat Reader free for multiple platforms and promoted PDF extensively, so it is the de facto standard for page description languages on the web these days.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 45.7 introduced one tool that can convert plain text into PostScript for printing. In general, if your printer isn't an old text-only printer and you want to be able to print text files, you'll need some sort of filter (or filters) to convert the text into something useful.

If your printer supports PostScript, tools like a2ps and enscript (Section 45.7) can do what you need. If your printer supports PCL or another printer language, you may want to add ghostscript to the mix. ghostscript can read PostScript and PDF and output correct representations to a variety of printers. Incidentally, ghostscript can also do a host of other useful things, like create PDFs from PostScript and the like.

Here's an example of using enscript, ghostscript, and lpr to print the background.txt file to my printer (an HP LaserJet 6L):

% enscript -2Gr background.txt -o background.ps

% gs -q -dNOPAUSE -sDEVICE=ljet4 -sOutputFile=background.lj4 background.ps -c quit

% lpr background.lj4

% rm background.lj4 background.ps

-2Gr tells enscript that I want two-up pages with fancy headers, and -o sends the output to background.ps (remember that enscript generates PostScript). -q tells gs to run quietly. -dNOPAUSE disables ghostscript's usual behaviour of pausing and prompting at the end of each page. -sDEVICE=ljet4 says to create output for a ljet4 device. -sOutputFile=background.lj4 redirects the output of ghostscript to background.lj4, and -c quit says to quit once background.ps is done. Then we use lpr to spool the now-ready output file, delete the temporary files, and we're all done.

Seems like sort of a pain, but it does show all of the steps needed to get that output to go to the printer properly. Section 45.17 shows how to arrange for most of that to be done for you by the spooler automatically.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 45.15 showed the steps necessary to convert plain text into something printable. Generally the steps involved are similar for a typeset source file, with perhaps an extra step or two.

troff generates PostScript by default in most installations these days, or it can be made to easily enough. GNU troff (groff ) can also generate PCL, DVI, and HTML by using the appropriate -T option.

T_EX generates DVI; the teT_EX package includes dvips to convert DVI into PostScript, dvilj4 to convert it into PCL, dvipdf to convert it into PDF, and several others.

HTML can be converted into PostScript using html2ps.

An example of using L^AT_EX, dvilj4, and lpr to print the article.tex file to my printer (an HP LaserJet 6L):

% latex article.tex

% dvilj4 article.dvi

% lpr article.lj

% rm article.lj article.dvi

This time it's slightly simpler than the example in Section 45.15, because the default options all do what we want. Even so, it can be made even simpler; Section 45.17 shows how.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Section 45.15 and Section 45.16 showed what sorts of steps are required to get files into a printable form. They seem tedious, however, and computers are really quite good at tedium, so how can we make the spooler do all this for us automatically?

There are a couple of options. One of the more well-known is apsfilter , which is a set of filter scripts designed to work with lpd to automatically convert incoming source files to an appropriate output format before dumping them to the printer. Extensive information is available at http://www.apsfilter.org, and apsfilter has its own automatic setup scripts, but I'll give a quick overview to give you an idea of what configuring lpd's filters looks like.

In Section 45.9, we used an input filter trick to print to a Samba printer by putting a if entry in the printcap for that printer. if stands for "input filter," and there are several other kinds of filters available in standard lpd, including a ditroff filter, a Fortran filter (!), and an output filter.

apsfilter installs itself as the input filter for any printer it manages, and looks at the source file. It decides based on a number of pieces of information what kind of source file it is, automatically processes it with the right set of programs, and poof, you have correct output coming out of your printer. There's a reason this kind of tool is called a "magic filter" (and why the title of this chapter says "Automagically"). Having a magic filter installed makes life so much easier.

If you look at your printcap once apsfilter is installed, you'll notice this entry (or something much like it):

lp|local line printer:\

     ...

     :if=/usr/local/sbin/apsfilter:\

     ...

That's all it takes to hook into lpd and tell the spooler to give apsfilter a shot at the text on the way through. apsfilter looks at the incoming file and its configuration for the printer queue and converts the source into the appropriate printer language using whatever filter or set of filters are needed.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The Common Unix Printing System (CUPS) is a full network-capable printing package available for a wide variety of Unix platforms. From their web page:

CUPS is available at:

http://www.cups.org/

CUPS provides a portable printing layer for UNIX-based operating systems. It has been developed by Easy Software Products to promote a standard printing solution for all UNIX vendors and users. CUPS provides the System V and Berkeley command-line interfaces.
CUPS uses the Internet Printing Protocol ("IPP") as the basis for managing print jobs and queues. The Line Printer Daemon ("LPD") Server Message Block ("SMB"), and AppSocket (a.k.a. JetDirect) protocols are also supported with reduced functionality. CUPS adds network printer browsing and PostScript Printer Description ("PPD") based printing options to support real-world printing under UNIX.

CUPS is headed towards becoming the Linux standard for printing, and it is an easy way to configure all your printing tools at once regardless of your platform. Visit their web page for extensive information.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

There are dozens of formats used for graphics files across the computer industry. There are tiff files, PICT files, and gif files. There are different formats for displaying on different hardware, different formats for printing on different printers, and then there are the internal formats used by graphics programs. This means that importing a graphics file from one platform to another (or from one program to another) can be a large undertaking, requiring a filter written specifically to convert from one format to the next.

Go to http://examples.oreilly.com/upt3 for more information on: netpbm

The netpbm package can be used to convert between a wide variety of graphics formats. netpbm evolved from the original Portable Bitmap Package, pbmplus, written by Jef Poskanzer. A group of pbmplus users on the Internet cooperated to upgrade pbmplus; the result was netpbm. netpbm has relatively recently seen some active development again on SourceForge, and its current home page is http://netpbm.sourceforge.net.

The idea behind pbm is to use a set of very basic graphics formats that (almost) all formats can be converted into and then converted back from. This is much simpler than having converters to and from each individual format. These formats are known as pbm, pgm, and ppm: the portable bitmap, graymap, and pixmap formats. (A bitmap is a two-dimensional representation of an image; a graymap has additional information encoded that gives grayscale information for each bit; a pixmap encodes color information for each bit.) The name pnm is a generic name for all three portable interchange formats (with the n standing for "any"), and programs that work with all three are said to be "anymap" programs.

The netpbm package contains well over a hundred conversion programs. There are three basic kinds of programs:

Programs that convert a graphics file to one of the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

TCP/IP networking is a part of the Open Systems Interconnection (OSI) Model. Much like you can string together lots of little single-purpose Unix tools to do complex tasks, the OSI Model is made up of specific single-purpose layers that work together. Each layer builds on the layers below. Layers 1 and 2 are concerned with hardware; physical standards such as required voltages and low-level protocols like Ethernet reside there. Layers 3 and 4 are networking layers, which this article introduces. Layers 5 through 7 are application layers, where networking interfaces such as BSD sockets and applications such as web browsers, telnet clients, and diagnostic tools live.

For most Unixes, the fundamentals of networking (once you get past the network device drivers) are the Layer 3 Internet Protocol (IP) and a Layer 4 protocol on top of it, either the Transport Control Protocol (TCP), the User Datagram Protocol (UDP), or the IP Control Message Protocol (ICMP). These four protocols are so commonly treated as one unit that you'll often see them referred to together as TCP/IP.

IP's job is to get small chunks of data, called packets, from one machine to another. It is a "best effort" protocol; that is, it makes its best effort to deliver each packet to the right host, and if it can't, it simply drops the packet on the floor. It may seem like losing bits of your data would be a bad thing, but it turns out that this feature is part of what allows the Internet to route traffic around problems; higher-level protocols and applications notice that packets are being dropped and resend them, sometimes through better routes.

IP identifies machines through IP addresses . Every machine that wants to communicate with another machine via TCP/IP must have a unique IP address, unless it's using Network Address Translation (NAT) ( Section 46.1). When you dial up your ISP with a modem, your ISP assigns you a dynamic IP address, good for that modem session. When you have a dedicated broadband connection, often your ISP will assign you a small block of static IP addresses to use as you like. Each ISP is in turn assigned large blocks of IP addresses for them to dole out to their users, and traffic on the Internet travels from ISP to ISP based on the addresses they hold.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

TCP/IP networking is a part of the Open Systems Interconnection (OSI) Model. Much like you can string together lots of little single-purpose Unix tools to do complex tasks, the OSI Model is made up of specific single-purpose layers that work together. Each layer builds on the layers below. Layers 1 and 2 are concerned with hardware; physical standards such as required voltages and low-level protocols like Ethernet reside there. Layers 3 and 4 are networking layers, which this article introduces. Layers 5 through 7 are application layers, where networking interfaces such as BSD sockets and applications such as web browsers, telnet clients, and diagnostic tools live.

For most Unixes, the fundamentals of networking (once you get past the network device drivers) are the Layer 3 Internet Protocol (IP) and a Layer 4 protocol on top of it, either the Transport Control Protocol (TCP), the User Datagram Protocol (UDP), or the IP Control Message Protocol (ICMP). These four protocols are so commonly treated as one unit that you'll often see them referred to together as TCP/IP.

IP's job is to get small chunks of data, called packets, from one machine to another. It is a "best effort" protocol; that is, it makes its best effort to deliver each packet to the right host, and if it can't, it simply drops the packet on the floor. It may seem like losing bits of your data would be a bad thing, but it turns out that this feature is part of what allows the Internet to route traffic around problems; higher-level protocols and applications notice that packets are being dropped and resend them, sometimes through better routes.

IP identifies machines through IP addresses . Every machine that wants to communicate with another machine via TCP/IP must have a unique IP address, unless it's using Network Address Translation (NAT) ( Section 46.1). When you dial up your ISP with a modem, your ISP assigns you a dynamic IP address, good for that modem session. When you have a dedicated broadband connection, often your ISP will assign you a small block of static IP addresses to use as you like. Each ISP is in turn assigned large blocks of IP addresses for them to dole out to their users, and traffic on the Internet travels from ISP to ISP based on the addresses they hold.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

After you've been dealing with Internet services for a while, you come to remember certain well-known port numbers off of the top of your head. SMTP (Section 46.8) is port 25, HTTP is port 80, and so on. However, unless your memory is far better than mine, you won't remember them all.

Luckily, that's part of what /etc/services is for. It's a database of well-known ports with symbolic names; any program that can take a port number should be able to take the appropriate symbolic name instead. If you want to make sure your SMTP server is up, the following two commands are equivalent:

% telnet localhost 25

% telnet localhost smtp

The definitive database of well-known ports is currently available at http://www.iana.org/assignments/port-numbers. On most Unixes, /etc/services is just a snapshot taken at the time that version of that Unix was released. When installing new services, often you'll want to tweak your local copy of /etc/services to reflect the new service, if it's not already there, even if only as a reminder.

The format of the /etc/services is simple:


               service name        

               port/protocol        

               aliases

Comments within the file start with a pound sign (#). As an example, a few common entries from /etc/services:

ftp-data         20/tcp    #File Transfer [Default Data]

ftp-data         20/udp    #File Transfer [Default Data]

ftp              21/tcp    #File Transfer [Control]

ftp              21/udp    #File Transfer [Control]

ssh              22/tcp    #Secure Shell Login

ssh              22/udp    #Secure Shell Login

telnet           23/tcp

telnet           23/udp

smtp             25/tcp    mail         #Simple Mail Transfer

smtp             25/udp    mail         #Simple Mail Transfer

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

ifconfig can be used to configure network devices ( Section 44.8), but it also can be used to see the current network device configuration. ifconfig -a is very useful for this. Here's some sample output on a FreeBSD machine:

% ifconfig -a

rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

        inet 192.168.1.1 netmask 0xffffffc0 broadcast 192.168.1.255

        inet 192.168.1.5 netmask 0xffffffff broadcast 192.168.1.255

        inet 192.168.1.6 netmask 0xffffffff broadcast 192.168.1.255

        inet 192.168.1.7 netmask 0xffffffff broadcast 192.168.1.255

        ether 0a:5c:da:a3:53:11

        media: autoselect (100baseTX <full-duplex>) status: active

        supported media: autoselect 100baseTX <full-duplex> 100baseTX 10baseT/UTP

        <full-duplex> 10baseT/UTP 100baseTX <hw-loopback>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384

        inet 127.0.0.1 netmask 0xff000000

This shows two network devices: rl0, which is an Ethernet card, and lo0, which is the loopback device. rl0's primary IP address is 192.168.1.1, and it has aliases (that is, it also answers to) 192.168.1.5 through 192.168.1.6. This also shows me that both network devices believe that they're actively sending and receiving packets (UP) and shows various options set on each device.

The output on Linux is slightly different, but similar enough to easily find the same information. Linux also adds a few statistics to its ifconfig output that otherwise require a netstat to see. Especially useful are packets received and transmitted:

eth0      Link encap:Ethernet  HWaddr 0a:5c:da:a3:53:11

          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:18999386 errors:28965 dropped:0 overruns:0 frame:28965

          TX packets:33955631 errors:0 dropped:0 overruns:0 carrier:0

          collisions:29132 txqueuelen:100

          RX bytes:1496731954 (1.3 GiB)  TX bytes:2477239809 (2.3 GiB)

          Interrupt:10 Base address:0xda00



lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:3924  Metric:1

          RX packets:107211318 errors:0 dropped:0 overruns:0 frame:0

          TX packets:107211318 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:2880669120 (2.6 GiB)  TX bytes:2880669120 (2.6 GiB)

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

ping is a very simple tool and often the first used to diagnose a network problem. ping sends one or more ICMP (Section 46.1) Echo Request messages to a particular IP address. If there is a machine at that IP address listening for ICMP messages (and no firewall filtering out packets in the middle), ping gets back Echo Reply messages, thus telling you that basic IP communication is functional between the two machines. If you can't ping something close by and you know you don't have a firewall (Section 46.12) filtering out your packets, it's generally not worth trying anything more complex; start looking for interfaces down or routing problems (Section 46.3) or, possibly, unplugged cables or the like.

traceroute does what you might expect from the name: it traces the route between your machine and another machine, using ICMP messages, and shows you each step of the way. Sometimes, when you can't get to another machine that's far away, you can use traceroute to see what's going on.

mtr stands for Matt's traceroute and is a more sophisticated traceroute. Not only does it show you each hop along the way, but it also sends multiple messages and gives you an ongoing display of latency at each hop. I use mtr instead of traceroute pretty exclusively. It's available at http://www.bitwizard.nl/mtr/, or your Unix may have a binary package of it available.

For serious network debugging, take a look at tcpdump and ethereal. tcpdump can take apart packets as they go by and show you what's going on in excruciating detail, and ethereal provides a nice GUI on top of tcpdump.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

inetd is the primary manager of Internet services on most Unix installations. Its job is to listen on a selection of ports (Section 46.1) and start up the appropriate server when a connection comes in. This frees servers that run under inetd from having to deal directly with networking issues and sockets.

inetd is configured via /etc/inetd.conf , which lists all the ports inetd should manage, the server associated with each port, and any special options for that server. For specific details, read the manpage, inetd.conf(5). As an example, here are a few fairly standard entries from inetd.conf on my FreeBSD system:

ftp     stream  tcp  nowait       root    /usr/libexec/ftpd      ftpd -l

telnet  stream  tcp  nowait       root    /usr/libexec/telnetd   telnetd

finger  stream  tcp  nowait/3/10  nobody  /usr/libexec/fingerd   fingerd -s

tftp    dgram   udp  wait         nobody  /usr/libexec/tftpd     tftpd /tftpboot

A common package included in many inetd distributions (and easily added to others) is called tcp_wrappers . tcp_wrappers allows you to create access rules to control incoming connections (generally stored in /etc/hosts.allow ) and deny connections from unauthorized hosts. This can be very handy even for machines behind a firewall (Section 46.12), as it provides extra security by guaranteeing that certain kind of connections will not be allowed into your machine. As an example, my home firewall allows SMTP (Section 46.8) and SSH (Section 46.6) connections in, but my hosts.allow denies connections from hosts that cannot be reverse resolved (Section 46.9), thus requiring a certain level of legitimacy before my machine will talk to a host.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

telnet was the original application for connecting to a remote machine via the Internet. (rsh was developed as a quick hack because telnet wasn't quite ready, and so became popular enough to be included in distributions going forward, but telnet was always supposed to be the "real" application.) In its normal mode, telnet connects to an inetd (Section 46.5)-managed daemon called telnetd, which manages the login process.

Unfortunately, the login process happens entirely in cleartext, as does all interaction with the remote shell program. Anyone tapping into the connection could get access to the user's password and thus gain illicit access to the remote system. To prevent this, Secure Shell (SSH) was developed. SSH uses Secure Sockets Layer (SSL), the same security mechanism that web browsers use. All interactions between your machine and the remote machine are encrypted, thus protecting your passwords and any other sensitive information. Its syntax is much like rsh's:

% ssh gabriel

               Logs into gabriel using your local username.

% ssh deb@bits.oreilly.com

               Logs into bits.oreilly.com using the login name deb.

% ssh michael ls /tmp

               Runs ls /tmp on michael.

% ssh deb@eli grep deb /etc/passwd

               Runs grep deb /etc/passwd on eli, using the login name deb.

SSL uses public key encryption, which means that connections are protected with operations based on a public/private key pair. Information encrypted with the public key can be decoded with the private key and vice versa. A server runs sshd , which, much like telnetd, accepts connections and manages the login process. (Unlike telnetd, sshd is generally not managed by inetd, because sshd's startup is complex and thus too slow to do every single time a connection is created. Because of this limitation, sshd has access rules much like tcp_wrappers' built in — generally by just linking with tcp_wrappers.) Each server has its own public/private key pair, allowing a user connecting to that server to verify its identity. This allows you to be sure that someone hasn't managed to redirect your connection to their machine instead (where they could collect your password, for example).

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Providing an anonymous FTP server allows anyone to anonymously download (and possibly upload) files. Normally, logging into an FTP server requires an account. Anonymous FTP creates an anonymous account and carefully limits its capabilities, so that you don't have to create a full account for everyone you might want to allow access to.

An anonymous FTP connection operates within a chroot , that is, an isolated area (see the manpage for chroot(8) and chroot(2) for more details on what a chroot is). A few basic things need to be provided within the chroot: a copy of ls, minimal versions of /etc/passwd (sans passwords) and /etc/group to allow ls to display files properly, and so forth.

Some platforms provide a simple anonymous FTP setup. RedHat has an RPM called anonftp-* that installs a proper chroot. FreeBSD's install tool can set up an anonymous FTP chroot for you. Check your platform documentation to see if it has a simple setup for you. Failing that, refer to CERT's guide on safely setting up anonymous FTP at http://www.cert.org/tech_tips/anonymous_ftp_config.html.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Email is one of the most well-known and commonly used Internet services. The core of Internet email is the Simple Message Transfer Protocol (SMTP), which defines a simple, extensible mechanism by which hosts can exchange mail messages. SMTP is spoken by programs known as Message Transfer Agents (MTAs); sendmail is the most well known of these and is included with the vast majority of Unixes. qmail , postfix, and exim are other common MTAs (I use qmail on all of my systems). Configuring an MTA generally involves telling it your default domain name for outgoing email, setting up whether it allows relaying and if so, under what limits (see below), possibly setting up spam filtering, and the like. It may also involve setting up MX records ( Section 46.9) for your domain(s).

Relaying is when an MTA allows someone to connect and send an email to an email address not served by that MTA. If you want to allow someone on your local machine or local subnet to send outgoing email via your MTA, this is a very good thing. An open relay allows anyone to send outgoing email, and this allows spammers to use your machine to send their spam. As you might guess, this is a Very Bad Thing. All MTAs have ways of limiting relaying so that local users can send email but spammers can't use your machine. Check your MTA's documentation, or take a peek at http://www.mail-abuse.org for more information.

Mail User Agents (MUAs or just UAs) provide the interface between users and MTAs. On Unix, these include programs such as mail, mailx, elm, and mutt, all of which work directly with the filesystem. Webmail clients are also MUAs, but they run under a webserver to provide networked access to mail. Often, though, you want to be able to use a MUA on another workstation that may or may not be a Unix machine, in which case you need some sort of MUA proxy to manage the mail and communicate with the remote MUA.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Usually, when you want to refer to a machine, you want to use its hostname, rather than having to remember its IP address ( Section 46.1). However, IP only understands IP addresses, not hostnames, so some mapping from hostname to IP address is necessary. /etc/hosts provides a simple mapping from hostname to IP address, but it has the disadvantage of being local to your machine. It would be impossible to maintain an /etc/hosts file that actually reflected the constantly changing reality of the Internet. (In fact, historically, /etc/hosts was a list of every single machine on the Internet, downloaded regularly from a central source. This system broke down when the number of hosts on the Internet surpassed a few hundred.)

The Domain Name Service (DNS) is a specification for a loosely coordinated, distributed database mapping host names to IP addresses. Generally, it's implemented by the Berkeley Internet Name Daemon (bind), running on hundreds of hosts. Each DNS server has authority over a small piece of the database, and coordination is accomplished through delegation. The root servers know which DNS servers have authority over the top-level domains (TLDs), such as .com, .net, .org, and so forth. Each of those DNS servers knows which DNS server has authority over each subdomain, and so on. DNS servers also cache information, so that a full, time-intensive search through the large distributed database isn't necessary every time you want to access a host's IP address.

DNS also stores other records, including Mail Exchanger (MX) records for routing mail (Section 46.8). MTAs use MX records when resolving where to send an email by looking up MX records on the domain for which the email is destined. Typically a DNS administrator creates an address record for mail. domain.com, points it at a machine configured to catch mail for domain.com, and then adds an MX record pointing to mail.domain.com on each host within

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most servers have one or more static IP addresses, which are generally set in one of the boot configuration files. However, it's not uncommon to have one or more workstations on your network, and its often convenient to configure their addresses in a central place. DHCP allows workstations to dynamically discover their IP addresses.

If you have a cable modem, it's quite possible you get your IP address via DHCP. Your cable provider has a DHCP server, and any machine you plug into your cable modem becomes a DHCP client, automatically getting an IP address from your provider's DHCP server. Section 46.11 describes NAT, which can let you run multiple machines on your home network in a case like this.

To run your own DHCP server, you need a DHCP daemon. isc-dhcpd is available at http://www.isc.org/products/DHCP/ and allows a variety of configurations. I have a variety of machines on my network at home, including servers with static IP addresses, workstations that use DHCP but always get the same IP address, and a few IP addresses dynamically allocated to random machines plugged into my network (handy for building a new machine or for friends visiting with their laptops).

Fixed dynamic addresses are extremely useful. Most of the normal workstations I have at home are configured to have fixed dynamic addresses: they get their IP addresses from the DHCP server, but the server recognizes each machine's Ethernet address (otherwise known as its MAC address) and hands out the same IP address each time. This allows me to have a centralized database of workstation addresses and makes configuration of those workstations trivial, while still giving me consistent IP addresses for all of my workstations.

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

For two separate networks to communicate, a gateway is needed. A gateway has two network interfaces (two network cards, a network card and a modem, or so forth) and routes packets between the two networks as appropriate. Routers and cable modems both function as gateways.

Unix machines can also function as gateways. There are several reasons to use your Unix machine as your gateway: it is generally more flexible than the built-in gateways in cable modems and DSL routers; it can function as a firewall ( Section 46.12); and if you have a limited number of IP addresses, it can perform Network Address Translation (NAT) for you.

NAT allows the machines on your LAN to use private addresses , that is, the address ranges set out in RFC1918 as reserved for private networks. These include 192.168.0.0 with netmask 255.255.0.0 (also known as 192.168.0.0/16), 172.16.0.0 with netmask 255.240.0.0 (also known as 172.16.0.0/12), and 10.0.0.0 with netmask 255.0.0.0 (also known as 10.0.0.0/8). Within the private network, you can have as many IP addresses as you need. The gateway runs a NAT server, which translates all the private addresses into a single public address (the address of the public side of the gateway) on the way out and back into the correct private addresses on the way back in. If you use DHCP ( Section 46.10) to configure your workstations, you can easily configure your gateway and NAT server to be your DHCP server also and hand out private addresses to your LAN.

Note that you can really only use private NAT for workstations. Servers that need to be externally accessible will need public IP addresses. If you are using a private network on your internal network, you can configure your NAT server to map a particular public address to a particular private address, allowing access to your server while still keeping the server behind your gateway/firewall. However, for a straightforward setup, each server will still need its own distinct public IP address, plus the main public IP address for the gateway. At the very least, you will need one public static IP address for the gateway; it is possible to configure

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Gateways ( Section 46.11) route packets from one network to another. Firewalls prevent some packets from being routed, based on a set of rules. Generally these rules are based on which direction the packet is going, to which port ( Section 46.1) it is destined or from which port it came, which protocol the packet is using (TCP, UDP, or ICMP for low-level protocols, though sometimes firewalls also recognize higher-level protocols like HTTP), and so forth.

A fairly standard firewall ruleset would allow outgoing packets from all machines on the LAN, disallow incoming packets that weren't part of an established connection (which allows machines on the LAN to establish connections going out, but keeps outsiders from establishing incoming connections), and then specifically allow things like incoming connections to port 25 (the SMTP (Section 46.8) port) on the mail server machine, ports 80 and 443 (the HTTP and HTTPS ports) on the web server machine, and port 22 (the SSH (Section 46.6) port) on any server that should be able to receive SSH logins.

Cable modems and DSL routers generally have simple firewalls built in; a Unix machine functioning as a gateway can also firewall and often has much more complex capabilities. Firewall software varies enough that detailed configuration of a firewall is beyond the scope of this book; things to look for include the documentation for ipfw, ipchains (Linux 2.2 kernel), or iptables (Linux 2.4 kernel).

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Often you have only dialup access but would like your home network to be able to access the Internet. A simple way to do this is to configure one Unix machine as a gateway (Section 46.11), with one side of the gateway your LAN and the other side the modem connection. If you then set up the modem connection to dial on demand, you have a simple way to share your connection between all of the machines on the LAN.

All that's required is that you set up your PPP connection ( Section 44.11), turn on PPP's NAT (Section 46.11) handling and then turn on gatewaying (Section 46.11). Make sure that all your LAN machines point to the gateway as their default gateway (handing out addresses via DHCP (Section 46.10) is an easy way to ensure this). Any attempt to access the Internet by any machine on the LAN will then cause your gateway to dial up your ISP, if the modem isn't currently connected.

Note that I said that you had to turn on NAT handling. A dialup almost always means that your dialup machine will be getting a dynamic address, and the only way to have multiple machines behind a dynamic address is NAT. Because this is so common, some PPP clients have NAT built in; no configuration is required and no separate natd needs to be run. NAT simply has to be enabled, generally with the -nat option. (Linux's pppd does not support NAT by itself. Read the masquerading HOWTO at http://www.linuxdoc.org/HOWTO/IP-Masquerade-HOWTO/ for more information on how to deal with NAT on Linux.)

— DJPH

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Too often, it seems, the discussion of operating systems devolves into accusations, recriminations, and hurt feelings. However, the reality of a heterogeneous computing environment makes cooperation among the various operating systems critically important. There are a number of ways that Unix machines can interact with and partipate in Windows networks. Many of those connections work in reverse, too, so that Windows users can begin to experience Unix without abandoning their preferred desktop. Polemics aside, operating systems are only a means to an end that is defined by your business. Fortunately, interoperability is becoming increasingly easier. The following sections will show some of the options available to you.

— JJ

Go to http://examples.oreilly.com/upt3 for more information on: Samba

Samba is an open source project that implements the Session Message Block (SMB) protocol, which is the core networking language of the Microsoft Windows family. Of course, the dominant networking protocol in Unix is the Transmission Control Protocol/Internet Protocol (TCP/IP). The challenge of the Samba project is to map SMB traffic onto TCP/IP networks. This is no small feat since SMB was designed for small, nonsegmented networks. Because all SMB network machine names exist in one global namespace, the practical size of an SMB network is quite limited. Although there are workgroups and NT domains (dolled-up workgroups with a domain controller), these groups don't partition a network in the same way that IP subnets do. Workgroups are simply an organizational grouping of machine names (although NT domains can also exercise some access control over the resources within their jurisdiction).

Despite these limitations, most offices these days have a very large installed base of Windows servers and workstations. With Samba, your Unix machine can participate in Windows file sharing and print services. In fact, Samba can replace Windows file and print servers in many cases. For the full reference on Samba (plus a good number of useful tips), pick up a copy of

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Too often, it seems, the discussion of operating systems devolves into accusations, recriminations, and hurt feelings. However, the reality of a heterogeneous computing environment makes cooperation among the various operating systems critically important. There are a number of ways that Unix machines can interact with and partipate in Windows networks. Many of those connections work in reverse, too, so that Windows users can begin to experience Unix without abandoning their preferred desktop. Polemics aside, operating systems are only a means to an end that is defined by your business. Fortunately, interoperability is becoming increasingly easier. The following sections will show some of the options available to you.

— JJ

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: Samba

Samba is an open source project that implements the Session Message Block (SMB) protocol, which is the core networking language of the Microsoft Windows family. Of course, the dominant networking protocol in Unix is the Transmission Control Protocol/Internet Protocol (TCP/IP). The challenge of the Samba project is to map SMB traffic onto TCP/IP networks. This is no small feat since SMB was designed for small, nonsegmented networks. Because all SMB network machine names exist in one global namespace, the practical size of an SMB network is quite limited. Although there are workgroups and NT domains (dolled-up workgroups with a domain controller), these groups don't partition a network in the same way that IP subnets do. Workgroups are simply an organizational grouping of machine names (although NT domains can also exercise some access control over the resources within their jurisdiction).

Despite these limitations, most offices these days have a very large installed base of Windows servers and workstations. With Samba, your Unix machine can participate in Windows file sharing and print services. In fact, Samba can replace Windows file and print servers in many cases. For the full reference on Samba (plus a good number of useful tips), pick up a copy of Using Samba from O'Reilly & Associates.

Samba consists mainly of two daemons and a host of supporting programs. The smbd daemon is responsible for making your machine's filesystem and printers available to a Windows network. The nmbd daemon handles the mapping of SMB machine names into the IP namespace and browsing other SMB resources. Some Unix systems, like Linux, are also able to mount other SMB drives onto their local filesystems using the smbmnt command.

Samba is available for all popular Unix platforms. The project web site, http://www.samba.org, is mirrored throughout the world, so you should be able to find a server near you. The current stable release of samba will be available as a link called

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The topic of security under Samba falls mainly into two categories: how to make the SMB server secure and how clients authenticate with the SMB server. Since the authentication issue is the thorniest, let's talk about it first.

In the [global] section of the smb.conf file, there is a directive called security that can take one of four values: share, user, server, or domain. Choosing share means that each shared resource has a set of passwords associated with it. Users must present one of those passwords to use the resource. User security requires users to provide a username and password to gain access to any of the shares. Samba can ask another SMB server to authenticate user credentials, instead of using local files, by selecting the server security setting. If you choose this security option, you will need to provide the

password

server

directive a space-separated list of NETBIOS machine names that will do the authentication. The last security option is domain. In this model, your machine joins an existing NT domain that does all the user credential authentication.

If you are new to Samba, your best bet is to use user security. The ugliest problem of Samba now rears its head: to use encrypted passwords or not to. The issue here is that older Windows clients (early Windows 95 and pre-SP3 NT 4.0) send user passwords over the network in clear text. The good news about clear text passwords is that Samba can use your system's /etc/passwd to authenticate users. All real accounts on your system will use their Unix username and password to connect to your SMB shares. The problems with this approach are:

Passwords can be easily snooped from the network.
Every SMB user requires a real account on your system.
Newer SMB clients will need to be patched to connect to your shares.

If the first two reasons don't scare you off using clear text passwords, the last reason is pretty daunting if you need to patch a lot of workstations. However, if you still want to go this route, you need to add the elements listed in Table 47-1 to each client's registry (using

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Modern versions of Samba come bundled with a web adminstration tool called swat. swat doesn't need any web server to run, but you will need to configure your system's inetd. As with any new service, you'll need to define a name and a port for it in /etc/services. For instance:

swat    901/tcp

You are now ready to make inetd serve swat. Add the following to /etc/inetd.conf :

swat  stream  tcp  nowait.400 root /path/to/samba/bin/swat swat

Now, restart inetd and point your web browser to http://localhost:901. You will be asked for the root username and password. If all is successful, you will see a screen that has seven navigation icons: home, globals, shares, printers, status, view, and password, as shown in Figure 47-1.

Figure 47-1: SWAT's globals page

swat will be on the globals page first. Here you can set the global directives, such as workgroup name and security type. There is online help, should an option not be immediately clear. Clicking on the shares icon shows you the services you are currently advertising and allows you to add more. Clicking on printers allows you to configure which printers you share with the SMB network. You can even restart the server from swat.

Third-party browsing tools are also available for Samba. Some of these, like smb2www, are web applications that show you the local SMB neighborhood. Others, like gsnhood and xSMBrowser, are X11 programs that work somewhat like the Windows Network Neighborhood application. One of the advantages of the X11 browsers is that they can be configured to allow users to mount SMB drives (if your Unix supports the smbfs filesystem). You will certainly be rewarded by searching the web for third-party SMB tools.

— JJ

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sharing Unix printers with a SMB network is pretty straightforward. You can use swat to select the printer you want to share. If your /etc/printcap is configured correctly, swat will allow you to select one of those printers from a drop-down menu. You will then be able to set access controls over that printer, as well as make the printer browsable. Be sure to click the Commit Changes button when you're finished to save your settings. If you're running the SMB daemons as standalone processes, you can restart them from the status section of swat. In any case, your printers won't be shared until the daemons are restarted.

Of course, you can also edit the smb.conf file directly. Here's how to share your default printer with the SMB network:

[lp]

        printable = Yes

        printer name = lp

        ; printing = BSD

        ; print command = /usr/bin/lpr -r %s

        path = /var/spool/samba

        guest ok = Yes

        read only = No

        comment = hp

The block should start off with the lpr queue name. The most important directive for print shares is printable, which identifies this block as defining a print share. The printer name needs to match the Unix printer queue name for the printer you wish to share. If you have defined a global directive printing (which defaults to BSD style print queues), you don't need to worry about explicitly telling Samba how to print to the queue. In the commented-out print command directive, the %s stands for the file you wish to print. The path directive defines the samba print spool directory. This directory needs to be writable by the smbd process. Both guest ok and read only directives are simple access controls. The comment block is self-explanatory.

After saving your changes and restarting the Samba servers (if needed), your shares should be browsable by the SMB network. From Unix, you can print to SMB printers with the following command:

$ smbclient //netbios-name/printer

smb: \> print filename-to-print

Here the smbclient program (described in more detail below) is used to connect to the SMB printer. Using the interactive shell, the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

From Unix, you can connect to SMB shares with the smbclient command. It provides an ftp-like interactive environment for transferring files between your Unix system and the SMB share. It also is an excellent tool for debugging your Samba setup. For instance, you can see what shares are available from an SMB host with the following:

$ smbclient -L //netbios-name -U SMB_username

The -L flag request the list of available shares from the machine specified by its NETBIOS name. You may optionally provide an SMB username name with the -U flag. If no explicit username is provided, your Unix account name is used.

Once you have found a directory share that interests you, you can "log in" to it:

$ smbclient //netbios-name/share -U SMB_username

smb: />

You will be prompted for the SMB password associated with whichever account you used. If successful, you will be at an interactive prompt. You may type ? or help to get all the options available to you. Use the get command to copy files from the remote host to your Unix machine and put to copy files in the other direction. Like ftp, Samba also provides the filename wildcard variants mget and mput to allow you to handle multiple files easily.

— JJ

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Go to http://examples.oreilly.com/upt3 for more information on: VNC

Virtual Network Computing (VNC) is an open source project from AT&T Labs in Cambridge, England. It is a client/server system that allows users to manipulate desktop environments remotely. There are VNC servers for Unix, Windows, and pre-MacOS X environments. The options for clients are even wider since there is a Java applet VNC client. This can be used on any system that supports a modern web browser and Java. There is also a native VNC client for Unix, Windows, and pre-MacOS X machines. VNC provides a platform-independent way to control a heterogeneous network from any client platform.

VNC provides a live picture of a desktop. When you move your mouse on the client end, the mouse also moves on the server. The VNC client gets a kind of "graphic diff" of the change on the remote desktop and applies that to its current notion of the desktop. As you might guess, VNC isn't ideal for high-performance video games, but it is very serviceable for system administration and development.

You can get either precompiled binaries or the source code at http://www.uk.research.att.com/vnc/download.html. If you choose to compile VNC from the source, you will need to get and unpack the tar archive from the above site. To build the source, change into the unpacked archive directory and type:

$ xmkmf

$ make World && cd Xvnc && make World

If the compile goes cleanly, change to root and install:

# make install

Setting up a VNC server on a Windows machine is fairly straightforward. Simply grab the appropriate binary from the VNC download page, unzip the archive, and run the SETUP.EXE program in the vncserver folder. VNC will create a folder in the Start menu in which you'll find the VNC server program. When started, this program will ask you for a password that clients will need to be able to use your Windows machine.

Connecting to any VNC server requires three things. The first is the server's hostname or IP address. The second is the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Sometimes you will need to use a Windows application that hasn't been ported to Unix. While you can buy an additional Windows machine just for that program, there are a few Unix solutions that will allow you access to the Windows environment from the comfort of X. While none of the solutions offered below have the performance of Windows running natively on dedicated hardware, each is worth mentioning.

What's the next best thing to having another machine run Windows? Having a virtual machine running Windows. VMWare, Inc., has produced software called vmware for Intel Linux that creates a virtual i386-class machine on which Windows can be installed. All your hardware is virtualized, so the virtual machine created is a somewhat slower clone of the host. Still, the performance is adequate for Office applications and development. vmware creates a private network on your machine so that, with Samba (Section 47.2), you can get to your Unix filesystem from your virtual Windows machine. You can get an evaluation copy at http://www.vmware.com.

If a virtual machine is overkill for your needs, you might want to look into the open source project called wine . A recursive acronym for Wine Is Not an Emulator, the wine project also runs only on Intel machines, and it tries to emulate the Windows API for Windows-native applications. This project has been under development for a long time and isn't quite ready for mission-critical applications yet. However, many Windows projects can mostly function under wine, including some video games, such as Blizzard's StarCraft. You will find more information about wine at http://www.winehq.com.

— JJ

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix users needing to access Windows applications will find that VNC is not a workable solution in all instances. The reason for this is that Windows operating systems were not designed to be multiuser; they do not allow multiple concurrent user sessions. When you have more than a few users needing to run a Windows application, such as Outlook to connect to corporate email, your options are to put a Windows PC on every desk, run Windows under a virtual machine, or set up Windows Terminal Services (WTS).

WTS is the current name of the multiuser software Microsoft provides with the Windows 2000 Server product family. Its former iteration was Windows NT 4.0 Terminal Server. Similar to VNC, WTS provides a Windows 2000 desktop to a connecting client, but does it in true multiuser fashion. Dozens of users can be connected to the same machine, running different processes, all independent of the other. However, WTS is only part of the solution for Unix users. This is because Microsoft only allows connections to a WTS server via the Remote Desktop Protocol (RDP) but doesn't provide any non-Windows clients that use RDP.

On the flip side, Citrix provides a Unix client program that can connect to a WTS server, but it only uses the Independent Computing Architecture (ICA) protocol. For that client to work, a server add-on product to WTS called Citrix Metaframe must be installed. Thankfully, Metaframe provides additional features to a WTS server besides ICA connectivity that helps to justify the additional cost.

One thing to be careful of when implementing a WTS solution is licensing. Microsoft is very strict in its rules about what machines can connect under which circumstances. Like tollbooths on the highway, Microsoft wants to get paid no matter how you get on, or which vehicle you're driving. To put licensing simply, you must have a Windows 2000 Server license for each server, a Windows 2000 Server Client Access License for each machine connecting to the server, a Terminal Services License for each machine actually using WTS, and, if you are using Office, each machine that runs Office off the WTS server must have a license. These are not concurrent licenses: if 50 machines are going to use Office at some point, all 50 must have licenses, not just the 10 that are connected at any given moment. Citrix licenses are in addition to Microsoft licenses but are thankfully more friendly. Citrix allows the use of concurrent licenses, which means 20 licenses could cover the needs of 50 users, if only 20 are going to be connected at a time. Full details about Microsoft licensing in a WTS environment can be found at

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Rather than being impregnable fortresses of steel, most computers are about as leaky as old wooden fishing boats. Though the press has focused primarily on Windows security violations in the last few years, Unix boxes are just as vulnerable and require as much, or more, effort to keep safe.

If your Unix box sits in your home, it is protected from unauthorized access, you live alone, and you never connect to the Internet, security probably isn't a concern for you. However, chances are your Unix box is fairly easy to access physically, and your system is most likely connected to the Internet through a modem or other network connection. In both these cases, this chapter and those that follow are of extreme interest to you.

Anytime you have a multiuser system, your account is vulnerable to others in the system and to anyone who might break into the system from outside your organization. The only way to protect accounts is to ensure that good account management practices are in place, such as removing accounts when people are no longer with the organization and using difficult-to-hack passwords, as well as making sure that sensitive data is protected by accidental or deliberate access.

For single-user systems, you'll want to make sure that someone can't accidentally or deliberately log into your machine at home or work. Chances are no one would try, but particularly if you have something such as Linux installed on a laptop, you're going to want to keep the snoops out.

More importantly, before you connect to the Internet, you have to know what you're doing with your system, particularly if you run applications such as web servers on your system. All you need is one harmful worm or virus, or to have a cracker break into your system, to have all your work and effort compromised.

The above areas of vulnerability — account, machine, and system — probably don't surprise you. But are you aware that you're vulnerable to yourself?

How many times have you accidentally deleted a file? Now, how many times have you deleted a file and not had backup in place? Security isn't just a protection against external intrusion. Used effectively, security is also an effective means to protect the system and the data and applications from internal error and blunder.