Chapter 9.1.2 already briefly touched upon the fact that sed is tasked with transforming text as a stream, not merely searching for it like grep. That capability is far more valuable than it appears at first glance: once we master sed, replacing a single configuration line across dozens of servers, cleaning thousands of log lines from sensitive data, or normalizing text file formats can be completed in a single command line, without needing to open a text editor at all. Sysadmins use sed almost daily to edit configuration files via automation scripts, while Developers use it to process build output or application logs inside a pipeline. This chapter thoroughly discusses the stream editor named sed: how it works, text substitution syntax along with its important flags, direct file editing (in-place), deleting and inserting lines, line addressing, up to real-world case studies on processing configuration files and logs. Basic Regex discussed in Chapter 9.3 will continue to be used throughout this chapter, so make sure the concepts are fresh before proceeding.
10.1 Concepts and How sed Works
This sub-chapter discusses what sed is briefly and how it processes text behind the scenes, just enough as a preparation before diving into substitution practice in 10.2.
10.1.1 What is a Stream Editor
sed stands for stream editor, a command-line tool that edits text non-interactively based on instructions we provide at the start, unlike interactive text editors such as nano or vim that require direct human operation. Instead of opening a file, editing manually, and then saving, we simply tell sed "replace word A with word B", and the entire file is processed automatically in an instant. This non-interactive nature is what makes sed so well-suited for use inside scripts, as it requires no human intervention in the middle of the process. sed was first developed at Bell Labs in the early Unix era, and to this day remains one of the most reliable text processing tools because it is almost always available by default on every Unix-like system.
10.1.2 Anatomy of a sed Command
The general form of a sed command consists of three parts: the program name, the script (instructions to run), and the file to process.
sed 'script' file.txtThe script itself usually consists of an address (optional, specifying which lines are affected by the instruction) and a command (what to do on that line), with the general pattern [address]command. This address concept is discussed in more detail in 10.3.3. To understand why sed works line by line, we need to know its execution cycle. Every time sed processes a file, it repeats the following four steps for each input line:
- Read one line from input, store it in a temporary working area called pattern space.
- Execute all commands in the script against the contents of that pattern space.
- Print the pattern space contents to stdout, unless the
-noption is used to suppress this behavior (discussed in 10.2.2). - Clear the pattern space, then repeat from step 1 for the next line until all input is exhausted.
The most important point of this cycle: sed by default never modifies the original file. Processed output is only printed to stdout, while the source file remains untouched, unless we explicitly use the -i option discussed in 10.3.1. This "safe by default" behavior is important to remember, as many beginners assume sed 's/a/b/' file.txt directly modifies file.txt, whereas that command only displays the processed result on the screen.
10.1.3 Checking the Installed sed Version
Before practicing, it is a good idea to check the sed variant installed on our system, because its behavior differs slightly across operating systems, especially regarding the -i option discussed in 10.3.2.
sed --versionOn Linux and WSL, the command above usually displays a first line containing sed (GNU sed) followed by a version number, because almost all Linux distributions and WSL use GNU sed. On macOS, the system default sed is BSD sed, which does not recognize the --version option at all, so the command above will display an error message like sed: illegal option -- -. This is not a sign of something broken, just proof that BSD sed does not support GNU-style long options. Users and Sysadmins working on macOS who want consistent behavior with Linux can install GNU sed via Homebrew (the Homebrew installation topic was touched upon in Chapter 1.3), which installs under the command name gsed to avoid conflicting with the system default sed.
brew install gnu-sed
gsed --version10.2 Text Substitution and Common Flags
Text substitution is the most common reason people use sed. This sub-chapter discusses the basic syntax of the s/// command along with the flags most frequently used in the field.
10.2.1 Basic Syntax of s///
The substitution command has the form s/pattern/replacement/, three parts separated by delimiter characters (usually forward slashes /). pattern is the regex searched for (regex rules are the same as grep in Chapter 9.3), and replacement is the text replacing it.
echo "hello world" | sed 's/world/bash/'The command above displays hello bash. One crucial thing to remember: without additional flags, s/// only replaces the first occurrence of the pattern in each line, not all occurrences.
echo "cat cat cat" | sed 's/cat/dog/'The result is dog cat cat, only the first word changes. The scope of "all occurrences" is only activated via the g flag, discussed in 10.2.2.
The forward slash delimiter in s/// can also be replaced with other characters, provided it is used consistently across all three positions. This is very helpful when the pattern or replacement itself contains many forward slashes, such as file paths, because without an alternative delimiter we would have to escape every forward slash with a backslash, which quickly makes commands hard to read.
sed 's/\/etc\/nginx\/nginx.conf/\/etc\/nginx\/nginx.conf.bak/' path_list.txt
sed 's#/etc/nginx/nginx.conf#/etc/nginx/nginx.conf.bak#' path_list.txtBoth commands above perform the same action, but the second line is much easier to read because it uses # as the delimiter, allowing slashes inside paths to be written as-is without escaping. Any character can be used as a delimiter as long as it does not appear within the pattern or replacement itself; other common choices besides # are | or ,.
10.2.2 Flags g, i, and p
These three flags are added after the third closing delimiter in the s/// command, and can be combined in a single command to control the scope and behavior of the substitution.
The g flag (global) causes sed to replace every occurrence of the pattern in a single line, not just the first.
echo "cat cat cat" | sed 's/cat/dog/g'The command above produces dog dog dog. Keep in mind that "global" here means global within a single line, because sed automatically processes every line in the entire file, so the g flag is purely about the number of occurrences per line, not file-wide scope. As an additional variation, sed also accepts numbers as flags, such as s/pattern/replacement/2 which replaces only the second occurrence, or the combination s/pattern/replacement/2g which replaces the second occurrence and onwards. This numerical combination is rarely used day-to-day, but is useful for specific cases when we only want to change a particular occurrence, not all of them.
The i flag makes pattern matching case-insensitive, ignoring uppercase and lowercase differences, just like the -i option in grep discussed in Chapter 9.2.3 although its placement differs (in sed, i is a flag after s///, not a separate command-line option).
sed 's/error/FAILED/gi' /var/log/app.logThe command above replaces error, Error, or ERROR simultaneously with FAILED. An important portability note: the i flag for s/// is a GNU extension, not part of the POSIX standard. This means this flag works normally in GNU sed (Linux, WSL), but is not recognized by macOS default BSD sed mentioned in 10.1.3, triggering errors like bad flag in substitute command. If a script needs to be portable to macOS without GNU sed installed, the alternative is to normalize text case first using tr '[:upper:]' '[:lower:]' as described in Chapter 9.4.4, before matching without the i flag.
The p flag prints the pattern space again after a successful substitution, separate from the automatic printing at the end of the cycle explained in 10.1.2. If used alone, the effect is that matching lines appear twice, once printed by the p flag and once more by the normal cycle. This flag becomes truly useful when combined with the -n option, which suppresses automatic printing at the end of the cycle, so only explicitly requested lines are displayed.
sed -n 's/error/FAILED/gp' /var/log/app.logThe command above displays only lines containing error (after being replaced with FAILED), while other non-matching lines are processed but not displayed at all. Combining -n with p is also frequently used without substitution at all, purely to display specific lines based on line addressing (discussed in 10.3.3), functioning similarly to grep but with more precise line position control.
sed -n '5,10p' /var/log/app.logThe command above displays lines 5 through 10 only, combining head and tail (Chapter 2.3) in a single command.
10.3 In-Place Editing and Line Addressing
This sub-chapter discusses how to modify files directly using sed, how to target specific lines via line addressing, as well as commands to delete and insert lines.
10.3.1 Option -i for In-Place Editing
As touched upon in 10.1.2, sed by default only prints processed results to stdout without altering the source file. The -i (in-place) option alters this behavior: substitution results are written directly back to the original file.
sed -i 's/production/staging/g' config.envThe command above directly modifies the contents of config.env, without displaying anything on the screen. Because it permanently alters files and cannot be undone via sed itself, good practice in the field is to always run the command without -i first to inspect the results on screen, before adding -i once confident that the pattern is correct. GNU sed also supports automatic backup via a suffix attached directly after -i without a space.
sed -i.bak 's/production/staging/g' config.envThe command above saves a copy of the original file as config.env.bak before writing changes, a simple safety net strongly recommended for Sysadmins when editing production configuration files via automation scripts. This suffix must be attached directly without a space (-i.bak, not -i .bak), because GNU sed will misinterpret .bak as the target file name to process if written separately.
10.3.2 Troubleshooting: GNU sed vs BSD sed
The most common pitfall between GNU sed and BSD sed (macOS default, mentioned in 10.1.3) lies in this -i option. GNU sed considers the backup suffix optional, allowing -i to be used alone without a backup, whereas BSD sed requires the suffix argument to always be explicitly written, even if empty.
# GNU sed (Linux, WSL), without backup
sed -i 's/production/staging/g' config.env
# BSD sed (macOS default), suffix argument required, empty means no backup
sed -i '' 's/production/staging/g' config.envIf the GNU-style command above is executed as-is on macOS with default sed, BSD sed will treat 's/production/staging/g' as the backup suffix, and config.env as the script, resulting in an error or unexpected behavior. This is a classic trap for Sysadmins and Developers writing scripts on Linux and running them as-is on macOS. The most practical solution for scripts that need to be portable across multiple platforms is to use GNU sed via gsed installed in 10.1.3, ensuring consistent -i behavior across all platforms without writing OS-specific conditional branching.
10.3.3 Line Addressing
Line addressing determines which lines are affected by a sed command, written before the command itself. Without an address, the command applies to every line in the entire file.
| Address Form | Meaning | Example |
|---|---|---|
N | Line number N | 3d deletes line 3 |
$ | Last line | $d deletes the last line |
/regex/ | Lines matching regex | /^#/d deletes lines starting with # |
address1,address2 | Line range from address1 to address2 | 2,5d deletes lines 2 through 5 |
address! | Negation, inverse of the address | /^#/!d deletes all lines except comments |
Line ranges can also use regex on both sides, useful when we do not know exact line numbers but know opening and closing patterns of a text block.
sed -n '/^server {/,/^}/p' nginx.confThe command above displays the entire text block starting from a line beginning with server { to a line containing closing }, a practical way to inspect a single configuration block without opening the entire nginx.conf file.
10.3.4 Deleting Lines with d
The d command deletes lines matching the address before it from the pattern space, so that line is not included in the output.
sed '/^$/d' notes.txtThe command above removes all empty lines from notes.txt, using the same ^$ pattern explained in Chapter 9.3.4 to match lines without any characters. The most common real-world combination is cleaning configuration files of both comment lines and empty lines in a single command.
sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_configThe -e option allows multiple scripts to be executed sequentially within a single sed command, here deleting comment lines first, then empty lines.
10.3.5 Inserting and Replacing Lines
The three commands a, i, and c add or replace entire lines, unlike s/// which only replaces partial text within a line.
| Command | Function |
|---|---|
a text | Inserts text as a new line after the line matching the address |
i text | Inserts text as a new line before the line matching the address |
c text | Replaces the entire line matching the address with text |
sed '/^upstream backend {/a\ server 127.0.0.1:8080;' nginx.confThe command above inserts a new line right after the line containing upstream backend {. The syntax above uses the single-line format supported by GNU sed; POSIX standard originally required a backslash followed by a real newline before the inserted text, a format more portable to various sed variants but more cumbersome to type directly on the command line.
sed '/^Listen 80$/c\
Listen 8080' httpd.confThe command above replaces the entire Listen 80 line with Listen 8080. The difference with s///: c replaces the line as-is entirely without needing to know its full exact contents beforehand via complex regex, simply matching its address.
10.4 Case Studies: Configuration Files and Logs
This concluding sub-chapter combines all techniques from 10.1 through 10.3 into two scenarios frequently faced by Sysadmins in the field: modifying configuration directives and sanitizing logs from sensitive data.
10.4.1 Enabling Configuration Directives
A common scenario when hardening servers (a topic discussed in depth in Chapter 23): ensuring a directive (setting line) in a configuration file is active with a specific value, whether that directive is currently disabled via a comment # or active with a different value.
sed -i.bak 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_configThe pattern ^#\?PermitRootLogin.* matches lines starting with PermitRootLogin, whether preceded by comment symbol # or not, thanks to \? meaning "preceding character is optional" in BRE dialect (the same metacharacter as in BRE vs ERE table in Chapter 9.3.3). Matching lines, regardless of their complete contents after directive name, are directly replaced entirely with PermitRootLogin no. The -i.bak option ensures the original file is stored as a backup before modification, a step that must be performed before editing crucial configuration files like sshd_config on production servers, as regex pattern mistakes on this file could lock out SSH access completely.
10.4.2 Obfuscating Sensitive Data in Logs
Application logs sometimes accidentally record sensitive data like passwords or tokens within URLs or parameters, a real security risk if those log files are shared with other teams for debugging or uploaded to third-party monitoring systems. If such log files are leaked or fall into the hands of an Attacker, passwords and tokens within them could be used directly to access real accounts, not merely a theoretical risk. sed can obfuscate such data using capturing groups and backreferences.
echo "GET /login?user=budi&password=secret123 HTTP/1.1" | sed -E 's/(password=)[^&]*/\1***/'The command above yields GET /login?user=budi&password=*** HTTP/1.1. The portion (password=) is a capturing group, a pattern fragment enclosed in parentheses so its content can be reused in the replacement section via \1 (referring to the first group content). The pattern [^&]* matches all characters after password= until encountering an & character or end of line, namely the password value itself, which is then replaced with *** while preserving the text password= as-is via \1. The -E option here is used so parentheses ( ) are immediately recognized as grouping metacharacters without escaping, consistent with ERE explanation in Chapter 9.3.3.
10.4.3 Troubleshooting: BRE vs ERE in sed
sed inherits the exact same regex dialects as grep: BRE (Basic Regular Expression) as default, ERE (Extended Regular Expression) via -E or -r option. The most common beginner error is writing ERE-style patterns without the -E option, causing metacharacters like +, ?, |, and ( ) to be treated as ordinary literal characters rather than metacharacters.
echo "color: red|blue" | sed 's/(red|blue)/COLOR/'The command above does not produce color: COLOR as might be expected. Because it is executed without -E, parentheses and vertical bar are treated literally, so the pattern only matches if text actually contains characters (red|blue) as-is, including parentheses. The fix, according to BRE vs ERE table in Chapter 9.3.3, offers two options: add -E, or manually escape each metacharacter in BRE.
echo "color: red|blue" | sed -E 's/(red|blue)/COLOR/'
echo "color: red|blue" | sed 's/\(red\|blue\)/COLOR/'Both lines above produce color: COLOR. The safest habit in the field: as soon as the required pattern involves grouping, alternation, or quantifiers like + and {n,m}, use -E right from the start, because the result is far easier to read compared to BRE version filled with backslashes. After mastering sed for line-by-line text modification, Chapter 11 continues to awk, a tool superior when processed data has column structures, such as CSV files or logs where each line needs to be broken down into multiple meaningful fields.