Advanced Text Processing: awk

Chapter 9.1.3 briefly mentioned that awk excels when processing structured column data, unlike grep which only searches for lines, and sed in Chapter 10 which modifies text line-by-line as a stream. This advantage makes awk the primary choice whenever a text line needs to be broken down into meaningful components, such as df -h output containing disk size columns, or exported sales report CSV files where each line contains date, product, and price fields. Sysadmins use awk almost daily to summarize logs and system command outputs into concise reports, while Developers use it to extract specific columns from build outputs or structured data files within pipelines. This chapter covers awk thoroughly: the pattern-action concept foundational to every awk program, accessing columns via $1, $2, and built-in variables like NF/NR, control structures like conditionals, loops, and functions within the awk language itself, up to practical real-world case studies for building reports and summaries from CSV files and logs. ERE regex covered in Chapter 9.3.3 will be used throughout this chapter, so make sure concepts are fresh before proceeding.

11.1 Concepts and Structure of an awk Program

This section briefly covers what awk is and how the basic structure of an awk program is composed, providing sufficient background before moving into field processing in section 11.2.

11.1.1 What is awk

awk is not merely a single command line tool like grep or sed, but an independent programming language designed specifically for processing text-based data, especially data with repeating row and column structures. Its name is an acronym derived from the surnames of its three original creators, Alfred Aho, Peter Weinberger, and Brian Kernighan, who developed it at Bell Labs around 1977, the same era that saw the birth of sed discussed in Chapter 10.1.1. The most fundamental difference from sed is that while sed processes text as single whole lines, awk automatically parses each line into individual fields upon reading, allowing direct references to the second or third column without writing manual regular expression extractions.

Several implementations of awk exist, and their behavior may vary slightly, similar to the GNU sed vs BSD sed comparison in Chapter 10.1.3. GNU Awk (shortened to gawk) is the most feature-rich implementation and is commonly found across Linux distributions. mawk, a lightweight and fast implementation, serves as the default awk on Debian-derived distributions such as Ubuntu. Meanwhile, macOS ships with the original implementation maintained by its author, known as the one true awk or BWK awk (Brian W. Kernighan's initials), as the system default. All three adhere to the POSIX Awk specification for core features, so basic examples in this chapter run consistently across all variants. However, advanced features specific to gawk extensions will be explicitly noted to highlight portability limits, discussed in detail in section 11.4.4.

11.1.2 Pattern and Action Anatomy

Every awk program consists of one or more pattern and action pairs, written in the following general form:

awk 'pattern { action }' file.txt

awk reads input line by line, where each line read is referred to as a record. For every record, awk evaluates whether the pattern matches. If it matches, the action block enclosed in curly braces is executed. Both pattern and action are technically optional, but at least one must be present. If pattern is omitted, the action executes for every line without exception.

awk '{print}' notes.txt

The command above displays the entire contents of notes.txt as-is, because without a pattern, the action {print} executes on every line, similar to the cat command discussed in Chapter 2.3. Conversely, if action is omitted, awk defaults to print $0, printing the matching line as-is, similar to how grep works in Chapter 9.2.

awk '/error/' /var/log/app.log

The command above displays every line in app.log containing the text error, because /error/ is a pattern form using regular expressions that matches against the entire line ($0). The regex dialect used by awk is fundamentally ERE (Extended Regular Expression), the same dialect used by grep -E or sed -E covered in Chapters 9.3.3 and 10.4.3, meaning metacharacters such as +, ?, |, and ( ) are recognized directly without escaping.

In addition to regular expression patterns, awk features two special patterns, BEGIN and END, which are not tied to any input lines. The BEGIN block executes exactly once before the first line is read, whereas the END block executes exactly once after all input lines have been processed. This combination is useful for initial setup and final summary reports, a pattern widely used in the case studies in section 11.4.

awk 'BEGIN {print "Processing started..."} /error/ {count++} END {print "Total error lines:", count}' /var/log/app.log

The program above combines three patterns in a single command line: BEGIN prints an opening message, the pattern /error/ increments the variable count whenever a line containing error is matched, and END prints the final total. One key distinction between awk and Bash is that variables in awk do not require prior declaration and automatically default to 0 or an empty string upon first use, enabling count++ to work directly without initial count=0 assignment inside the BEGIN block.

11.1.3 Checking Installed awk Version

Before working with further examples, it is helpful to check the specific awk variant installed on your system, as advanced feature availability varies between implementations as noted in 11.1.1.

awk --version

If gawk is installed, the command above outputs a line displaying GNU Awk along with its version number. Note that the --version option is not part of the official POSIX Awk specification, so it is not guaranteed to be recognized across all implementations. mawk, the default on Debian-based distributions like Ubuntu, recognizes this option (derived from its internal -W version option) and displays version and copyright details, though formatted differently than gawk. The difference becomes noticeable on macOS BWK awk mentioned in 11.1.1: this implementation does not recognize long GNU-style options at all, treating --version as an unrecognized option and typically throwing an error that no program was supplied rather than displaying a version. If running the command results in an ambiguous error message, the safest approach to identify the installed implementation is checking the binary path directly.

command -v awk
readlink -f "$(command -v awk)"

Users and Sysadmins working on macOS who require full gawk features consistent with Linux environments can install it via Homebrew (Homebrew installation was introduced in Chapter 1.3). Unlike GNU sed which installs as gsed on Homebrew to avoid name conflicts as discussed in Chapter 10.1.3, the gawk package installs directly under the command name gawk, since no native binary named gawk exists on macOS to conflict with.

brew install gawk
gawk --version

11.2 Field Processing and Built-in Variables

This section explains how awk breaks down lines into individual columns called fields, along with the most common built-in variables used to inspect and manipulate them.

11.2.1 Fields and Records: $0 to NF

By default, awk splits lines into fields using whitespace (spaces and tabs), allowing direct references to specific columns without calculating character positions manually as with ${var:offset:length} in Chapter 9.4.2.

df -h | awk '{print $1, $NF}'

The command above prints the first column (Filesystem) and the last column (mount point) from the output of df -h, a utility introduced in Chapter 17. The $NF notation is a practical technique because NF holds the total field count for the current record, causing $NF to dynamically resolve to the final field regardless of line length or column count. One edge case to keep in mind is that NF can vary line by line within the same file. For example, the header row of df -h contains the multi-word title "Mounted on", resulting in one more field than subsequent data rows containing single-word paths. When fields contain spaces, do not assume NF remains constant across every row.

11.2.2 Record Counters: NR and FNR

NR increments by one for every new record read by awk, continuing sequentially across multiple files. FNR behaves similarly, but resets to 1 whenever awk begins reading a new file.

awk '{print FNR, NR, $0}' app-1.log app-2.log

If app-1.log contains 5 lines, the first line of app-2.log displays an FNR value of 1 (since a new file started), but an NR value of 6 (continuing count from the previous file). Understanding this distinction is essential when batch processing multiple files where file tracking is needed. The most frequent real-world use of NR is skipping header rows in CSV files, as the first row typically holds column headers rather than data rows.

awk 'NR > 1 {print}' data.csv

The command above prints all rows in data.csv except the first line. The expression NR > 1 serves as a comparison pattern, a pattern form detailed further in section 11.3.1.

11.2.3 Field Separators: FS, -F, and OFS

The default field separator in awk has unique behavior: when FS is set to a single space character (its default value), awk treats contiguous sequences of spaces and tabs as a single delimiter, while ignoring leading and trailing whitespace. This behavior keeps the df -h output clean in section 11.2.1 despite variable column widths.

For files using non-whitespace delimiters, such as comma-separated CSV files or colon-separated /etc/passwd files, set FS explicitly using the -F command-line option.

awk -F: '{print $1}' /etc/passwd

The command above outputs system user account names, taking the first field of each /etc/passwd line separated by colons from fields like UID and home directory. For CSV files, the same pattern applies using a comma separator.

awk -F',' '{print $2}' sales.csv

FS can also accept multi-character patterns. When set to multiple characters, awk interprets FS as an ERE regular expression rather than a literal character, which is useful when parsing input containing multiple delimiter types.

awk -F'[,;]' '{print $1}' mixed_data.txt

The command above splits fields using either a comma or a semicolon, matching either delimiter using character class [,;] as covered in Chapter 9.3.2.

While FS controls input parsing, OFS defines the output delimiter used when awk reconstructs $0 for printing. An important rule to note: OFS takes effect only when $0 is reconstructed, such as when a field value is modified. Printing $0 directly without modifying fields preserves the original input delimiters rather than applying OFS.

awk 'BEGIN {FS=","; OFS="|"} {$1=$1; print}' sales.csv

The assignment $1=$1 above reassigns the first field to itself. While seemingly a no-op, this operation forces awk to rebuild $0 using the updated OFS setting. As a result, comma-delimited input is printed with | delimiters, providing a straightforward way to convert file delimiters without using sed.

11.3 Conditionals, Loops, and Functions

This section explores control structures inside the awk language, covering conditional branching, loops, built-in functions, and user-defined functions, all placed inside the { action } block introduced in section 11.1.2.

11.3.1 if-else Conditionals and Comparison Operators

The if/else syntax in awk follows C language conventions, differing from Bash if/then/fi syntax discussed in Chapter 6.1.

awk -F',' 'NR > 1 {
    if ($3 > 10) {
        print $2, "high sales"
    } else {
        print $2, "low sales"
    }
}' sales.csv

Comparison operators follow C standards: ==, !=, <, >, <=, >=, combined with logical operators && (AND), || (OR), and ! (NOT), matching the logical operations in Bash covered in Chapter 6.4.2. For explicit regex matching within conditionals, awk provides the ~ (matches) and !~ (does not match) operators.

awk -F',' '$2 ~ /^Coffee/ {print}' sales.csv

The command above prints lines where the second field begins with Coffee, using the same ERE regular expressions as pattern /error/ in section 11.1.2, but scoped specifically to one field via the ~ operator instead of matching against the full line $0.

11.3.2 for and while Loops

awk supports for and while loops using C syntax, distinct from the for item in list syntax common in Bash (Chapter 7.1).

awk '{
    for (i = 1; i <= NF; i++) {
        print "Field", i, "=", $i
    }
}' notes.txt

The program above iterates through every field on each line, using NF as the loop bound and $i to dynamically reference fields via index variable i instead of static positional references like $1. Dynamic field evaluation allows awk to process files with variable column counts efficiently.

awk also supports while loops for condition-based iteration, and do-while loops when the loop body must execute at least once before testing the evaluation condition. In practice, for loops are most common since most awk iterations operate over known field counts or array elements.

11.3.3 Built-in Functions and User-Defined Functions

awk provides standard built-in functions for string manipulation and numeric operations, summarized in the table below.

awk -F',' '{print toupper($2), length($2)}' sales.csv

The command above prints product names in uppercase alongside their character counts, chaining two built-in functions. The sub and gsub functions operate similarly to the s/// command in sed discussed in Chapter 10.2.1, but are invoked as functions rather than command syntax.

echo "cat cat cat" | awk '{gsub(/cat/, "dog"); print}'

The command above outputs dog dog dog because gsub without a third target argument operates on $0 by default, replacing all occurrences similar to sed 's/cat/dog/g'.

You can also define custom user-defined functions using the function keyword, conceptually similar to Bash functions in Chapter 8, but using awk syntax.

function percentage(part, total) {
    return (part / total) * 100
}

BEGIN {
    print percentage(25, 200)
}

The program above defines a percentage function that accepts two parameters and returns the percentage value using return, matching return value concepts in Chapter 8.3. A standard awk idiom to note: extra parameters added to a function signature that are omitted during call invocation act as local variables. This replaces local variable keywords (like local in Bash, Chapter 8.4) which are not present in awk.

function mark_size(val,    result) {
    if (val > 100) {
        result = "large"
    } else {
        result = "small"
    }
    return result
}

Notice the extra spaces before result in the parameter list above. This convention distinguishes positional call parameters (val) from internal local workspace variables (result), serving as a visual indicator recommended by the awk community.

11.4 Case Studies: CSV Reports and Log Analysis

This concluding section combines techniques from sections 11.1 through 11.3 to construct analytical reports from CSV files, a common task for Sysadmins and Developers summarizing dataset information without spreadsheet software. Consider the following sales.csv file:

date,product,quantity,price
2026-08-01,Coffee,10,25000
2026-08-01,Tea,5,15000
2026-08-02,Coffee,8,25000
2026-08-02,Bread,12,12000
2026-08-03,Tea,7,15000

11.4.1 Summing and Averaging Numeric Columns

A common tabular data task is calculating column totals, such as computing total revenue across transactions.

awk -F',' '
NR > 1 {
    total += $3 * $4
    count++
}
END {
    printf "Total revenue: %d\n", total
    printf "Average per transaction: %.2f\n", total / count
}' sales.csv

The program skips the header line using NR > 1 as explained in 11.2.2, multiplies quantity ($3) by price ($4) for each line, and accumulates the value into total. The END block formats output using printf, matching Bash printf formatting from Chapter 5.1, including format specifiers like %d for integers and %.2f for floating-point values rounded to two decimal places. For the sample data above, total revenue is 774000 and average revenue per transaction is 154800.00.

11.4.2 Category Aggregation using Associative Arrays

Another frequent requirement is grouping metrics by category, such as calculating total revenue per product. awk provides associative arrays, which use strings as keys rather than incremental numeric indices, similar to declare -A in Bash (Chapter 4.2). In awk, arrays are implicitly associative without requiring explicit declarations.

awk -F',' '
NR > 1 {
    revenue[$2] += $3 * $4
}
END {
    for (product in revenue) {
        printf "%-10s Rp%d\n", product, revenue[product]
    }
}' sales.csv

The script uses the product name ($2) as the key for the revenue array, automatically aggregating total revenue for matching product names. The loop for (product in revenue) iterates over assigned array keys. For the sample data above, output consists of three summary lines: Coffee with 450000 revenue, Tea with 180000, and Bread with 144000. Note that POSIX Awk does not guarantee key iteration order for for...in loops, meaning iteration order can vary between executions or awk implementations. If ordered output is required, pipe output to sort (Chapter 5.3) as a secondary processing step.

11.4.3 Log Analysis Case Study: Counting HTTP Status Codes

Associative array aggregation applies to semi-structured log files as well as clean CSV files. Sysadmins frequently inspect web server access logs, such as Apache or Nginx logs formatted under the Common Log Format, shown in the single line example below:

192.168.1.10 - - [30/Aug/2026:10:15:32 +0700] "GET /index.html HTTP/1.1" 200 512

Although log lines are not comma-separated, default space delimiter parsing via FS as covered in 11.2.3 can split the fields. Because timestamps are enclosed in brackets containing spaces, and HTTP requests are wrapped in quotes containing spaces, these fields split into multiple elements. Under standard logging formats, field nine ($9) contains the HTTP response status code, and field ten ($10) contains response byte size.

awk '{status[$9]++} END {for (code in status) print code, status[code]}' access.log

This program uses the array aggregation pattern from section 11.4.2, indexing by HTTP status code rather than product name. For a log file containing three 200 statuses, one 404, and one 500, the command outputs three summary lines showing status codes alongside occurrence counts. Sysadmins use these one-liner scripts for rapid service health checks, such as counting 5xx server error counts in log files without opening dedicated monitoring tools. Further analysis of server logs, including journalctl, is covered in Chapter 19.

11.4.4 Troubleshooting: awk Portability

As discussed in section 11.1.1, the three common awk implementations (gawk, mawk, and macOS BWK awk) support standard POSIX Awk core functionality used throughout this chapter: pattern-actions, field variables, NF/NR, conditionals, loops, associative arrays, and core functions like substr and gsub. However, gawk includes non-POSIX extensions, such as gensub() for backreference replacements, strftime() for date formatting, or length(array) for direct array element counts. Scripts using these extensions will fail on mawk or BWK awk with undefined function errors.

When writing portable scripts intended across multiple systems with varied awk implementations, rely on POSIX Awk standard features covered in this chapter to ensure cross-platform compatibility. If gawk extensions are required, ensure gawk is installed on target machines and execute using the gawk binary name explicitly rather than awk, since the awk binary alias on target systems may point to non-GNU implementations.

Having covered grep, sed, and awk as core text processing utilities, Chapter 12 explores Bash script argument handling, covering positional parameters and option parsing via getopts.