Awk process file

The moral is to choose your data layout and separator characters carefully to prevent such problems. If the data is not in a form that is easy to process, perhaps you can massage it first with a separate awk program.

Fields are normally separated by whitespace sequences spaces, TABs, and newlines , not by single spaces. Two spaces in a row do not delimit an empty field. The default value of the field separator FS is a string containing a single space, " ". If awk interpreted this value in the usual way, each space character would separate fields, so two spaces in a row would make an empty field between them. The reason this does not happen is that a single space as the value of FS is a special case—it is taken to specify the default manner of delimiting fields.

If FS is any other single character, such as "," , then each occurrence of that character separates two fields. Two consecutive occurrences delimit an empty field. If the character occurs at the beginning or the end of the line, that too delimits an empty field. The space character is the only single character that does not follow these rules. The previous subsection discussed the use of single characters or simple strings as the value of FS. More generally, the value of FS may be a string containing any regular expression.

In this case, each match in the record for the regular expression separates fields. For example, the assignment:. For a less trivial example of a regular expression, try using single spaces to separate fields the way single commas are used. FS can be set to "[ ]" left bracket, space, right bracket.

This regular expression matches a single space and nothing else see Chapter 3. However, when the value of FS is " " , awk first strips leading and trailing whitespace from the record and then decides where the fields are. For instance, study this pipeline:. The first print statement prints the record as it was read, with leading whitespace intact. There is an additional subtlety to be aware of when using regular expressions for field splitting. Or is each field separator a new string?

It turns out that different awk versions answer this question differently, and you should not rely on any specific behavior in your programs. There are times when you may want to examine each character of a record separately. This can be done in gawk by simply assigning the null string "" to FS.

In this case, each individual character in the record becomes a separate field. Traditionally, the behavior of FS equal to "" was not defined.

In this case, most versions of Unix awk simply treat the entire record as only having one field. In compatibility mode see Command-Line Options , if FS is the null string, then gawk also behaves this way.

FS can be set on the command line. Use the -F option to do so. The latter option -f specifies a file containing an awk program. The value used for the argument to -F is processed in exactly the same way as assignments to the predefined variable FS. Any special characters in the field separator must be escaped appropriately. The following a university, and the first three digits of their phone numbers:. This demonstrates why you have to be careful in choosing your field and record separators.

Perhaps the most common use of a single character as the field separator occurs when processing the Unix system password file. On many Unix systems, each user has a separate entry in the system password file, with one line per user. The information in these lines is separated by colons. A password file entry might look like this:.

The following program searches the system password file and prints the entries for users whose full name is not indicated:. According to the POSIX standard, awk is supposed to behave as if each record is split into fields at the time it is read. In particular, this means that if you change the value of FS after a record is read, the values of the fields i. However, many older implementations of awk do not work this way. Instead, they defer splitting the fields until a field is actually referenced.

The fields are split using the current value of FS! This behavior can be difficult to diagnose. The following example illustrates the difference between the two methods:. It is important to remember that when you assign a string constant as the value of FS , it undergoes normal awk string processing. Fields are separated by runs of whitespace.

Leading and trailing whitespace are ignored. This is the default. Fields are separated by each occurrence of the character. Multiple successive occurrences delimit empty fields, as do leading and trailing occurrences. The character can even be a regexp metacharacter; it does not need to be escaped. Fields are separated by occurrences of characters that match regexp. Leading and trailing matches of regexp delimit empty fields. Each individual character in the record becomes a separate field. It has no effect when FS is a single character, even if that character is a letter.

Thus, in the following code:. If you really want to split fields on an alphabetic character while ignoring case, use a regexp that will do it for you e. This section discusses an advanced feature of gawk. If you are a novice awk user, you might want to skip it on the first reading.

For example, data of this nature arises in the input for old Fortran programs where numbers are run together, or in the output of programs that did not anticipate the use of their output as input for other programs. An example of the latter is a table where all the columns are lined up by the use of a variable number of spaces and empty fields are just spaces. Each number specifies the width of the field, including columns between fields.

If you want to ignore the columns between fields, you can specify the width as a separate field that is subsequently ignored. It is a fatal error to supply a field width that is not a positive number. The following data is the output of the Unix w utility. The following program takes this input, converts the idle time to number of seconds, and prints out the first two fields and the calculated idle time:. Another possibly more practical example of fixed-width input data is the input from a deck of balloting cards.

In some parts of the United States, voters mark their choices by punching holes in computer cards. These cards are then processed to count the votes for any particular candidate or on any particular issue.

Because a voter may choose not to vote on some issue, any column on the card may be empty. Of course, getting gawk to run on a system with card readers is another story! Assigning a value to FS causes gawk to use FS for field splitting again.

Normally, when using FS , gawk defines the fields as the parts of the record that occur in between each field separator. In other words, FS defines what a field is not , instead of what a field is. However, there are times when you really want to define the fields by what they are, and not by what they are not.

The most notorious such case is so-called comma-separated values CSV data. Many spreadsheet programs, for example, can export their data into text files, where each record is terminated with a newline, and fields are separated by commas. The problem comes when one of the fields contains an embedded comma. In such cases, most programs embed the field in double quotes. The FPAT variable offers a solution for cases like this.

The value of FPAT should be a string that provides a regular expression. This regular expression describes the contents of each field. Writing this as a string requires us to escape the double quotes, leading to:. A straightforward improvement when processing CSV data of this sort would be to remove the quotes when they occur, with something like this:. Some programs export CSV data that contains embedded newlines between the double quotes.

As written, the regexp used for FPAT requires that each field contain at least one character. Finally, the patsplit function makes the same functionality available for splitting regular strings see String-Manipulation Functions.

To recap, gawk provides three independent methods to split input records into fields. In some databases, a single line cannot conveniently hold all the information in one entry. They match before and after the document has been processed.

Each of the expanded sections are optional. In fact, the main action section itself is optional if another section is defined. For example, you can do things like this:. This was easy because you were looking for the beginning of the entire line.

What if you wanted to find out if a search pattern matched at the beginning of a field instead? You can tell awk to only match at the beginning of the second column by using this command:. This introduces a few new concepts. Using this, you can combine an arbitrary number of conditions for the line to match. You can use awk to process files, but you can also work with the output of other programs. You can use the awk command to parse the output of other programs rather than specifying a filename.

For example, you can use awk to parse out the IPv4 address from the ip command. The ip a command displays the IP address, broadcast address, and other information about all the network interfaces on your machine.

To display the information for the interface called eth0 , use this command:. You can use awk to target the inet line and then print out just the IP address:. This splits the line inet The operators are grouped with parentheses. The break statement immediately exits from an enclosing while or for.

To begin the next iteration, use the continue statement. The next statement instructs awk to skip to the next record and begin scanning for patterns from the top. The exit statement instructs awk that the input has ended.

Note: The awk tool allows users to place comments in AWK programs. Comments begin with and end at the end of the line. Inserting a pattern in front of an action in awk acts as a selector. The selector determines whether to perform an action or not. The following expressions can serve as patterns:. Note: Learn how you can search for strings or patterns with the grep command. Regular expression patterns are the simplest form of expressions containing a string of characters enclosed in slashes.

It can be a sequence of letters, numbers, or a combination of both. In the following example, the program outputs all the lines starting with "A". If the specified string is a part of a larger word, it is also printed. Another type of awk patterns are relational expression patterns.

A range pattern is a pattern consisting of two patterns separated by a comma. Range patterns perform the specified action for each line between the occurrence of pattern one and pattern two. The pattern above instructs awk to print all the lines of the input containing the keywords "clerk" and "manager". The Overflow Blog. Podcast Making Agile work for data science. Stack Gives Back Featured on Meta. New post summary designs on greatest hits now, everywhere else eventually.

Visit chat. Linked See more linked questions. Related Hot Network Questions. Question feed. Stack Overflow works best with JavaScript enabled.