sed

What is actually sed ?

The sed full form is a “Stream EDitor”. This sed command  is UNIX  utility. A “non-interactive” text editor that could be called from the UNIX command line. It input text flows through the program, is modified, and is directed to standard output. 

An Example: The following sentence is input to the sed program:

echo "Install and  ruining the WebLogic for a Middleware application operation." |

       sed 's/ruining/running/‘

Install and  running the WebLogic for a Middleware application operation..

Why you need to use “sed”?

Eliminate the tedium of routine editing tasks! (find, replace, delete, append, insert)

                … but your word processor can already do that right? It is Wrong! Because  the sed is extremely powerful and it comes with every UNIX flavor operating system in the world!

Hope now you will understand why you need sed with the following three cases:

1.       To edit files too large for comfortable interactive editing;

2.       To edit any size file when the sequence of editing commands is too complicated to be comfortably typed in interactive mode.

3.       To perform multiple `global' editing functions efficiently in one pass through the input.

How does sed Works?

The sed Architecture is as follows:

Pattern and Hold spaces

Pattern space: Workspace or temporary buffer where a single line of input is held while the editing commands are applied

Hold space: Secondary temporary buffer for temporary storage only

Let me put it in program sudo code way:

While (read line){

                1 )   reads an input line from STDIN or

a given file, one line at a time, into the   pattern space.

                Pattern Space = a data buffer - the “current text” as it’s being edited

                2)  For each line, sed executes a series of editing commands issued by user on the pattern space.

                3) Writes back the pattern space to STDOUT.

}

Example 2:

echo “Vasu enjoys hiking and Raghav enjoys skiing” |sed –e ‘s/skiing/hiking/g; s/hiking/biking/g’

1 ) Here the sed reads  the line “Vasu enjoys hiking and Raghav enjoys skiing” and executed  the first ‘substitute’ command.

 The resulting line – in the pattern space: 

                        “Vasu enjoys hiking and Raghav enjoys hiking”

 2) Then the second substitute command is executed on the line in the pattern space, and the result is :

        “Vasu enjoys biking and Raghav enjoys biking”

3) finally, the result is written to standard output.

Since only a few lines of the input reside in memory at one time, and no temporary files are used, the effective size of file that can be edited is limited only by the requirement that the input and output fit simultaneously into available on the hard disk.

 Invoking sed Commands

$ sed [-e script] [-f script-file] [-n] [files...]

-e an "in-line" script, i.e. a script to sed execute given on the command line. Multiple command line scripts can be given, each with an -e option.

-n by default, sed writes each line to stdout when it reaches the end of the script (being whatever on the line)  this option prevents that. i.e. no output unless there is a command to order SED specifically to do it

-f             read scripts from specified file, several -f options can appear

Files are the files to read, if a "-" appears, read from stdin,if no files are given, read also from stdin

Different ways to Invoke Sed:

sed –e 'command;command;command' input_file                   see results

sed –e 'command;command;command' input_file > output_file            save results

.... | sed –e 'command;command;command' | ....                use in a pipeline

sed -f sedcommands input_file > output_file            commands are in file somewhere else

1.       sed commands are usually on one line

2.       if we want more (multi-line commands), then we must end the first line with an `\'

3.       if a command is one line only, it can be separated by a `;‘

4.       if it is a multi-line, then it must contain all of its line (except the first) by themselves

5.       on command line, what follows a `-e' is like a whole line in a sed script

Editing commands

 

                                a                              append

                                c                              change lines

                                d                             delete lines                       

                                i                               insert

                                p                             print lines

                                s                              substitute

 

Syntax for these commands is a little strange because they must be specified on multiple lines

append                [address]a\

                                                text

insert                    [address]i\

                                                text

change                 [address(es)]c\

                                                text

append/insert for single lines only, not range

 

 

Append and Insert

Append places text after the current line in pattern space

Insert places text before the current line in pattern space

Each of these commands requires a \ following it.
text must begin on the next line.

If text begins with whitespace, sed will discard it
unless you start the line with a \

Example:

                /<Insert Text Here>/i\

       Line 1 of inserted text\

       \        Line 2 of inserted text

         would leave the following in the pattern space

                Line 1 of inserted text

                 Line 2 of inserted text

       <Insert Text Here>

Change

Unlike Insert and Append, Change can be applied to either a single line address or a range of addresses

When applied to a range, the entire range is replaced by text specified with change, not each line

Exception: If the Change command is executed with other commands enclosed in { } that act on a range of lines, each line will be replaced with text

No subsequent editing allowed

 

sed Change Examples

Remove mail headers, ie; the address specifies a range of lines beginning with a line that begins with From until the first blank line.

The first example replaces all lines with a single occurrence of <Mail Header Removed>.

The second example replaces each line with <Mail Header Removed>

Using !

If an address is followed by an exclamation point (!), the associated command is applied to all lines that don’t match the address or address range

Examples:

                1,5!d would delete all lines except 1 through 5

                /black/!s/cow/horse/ would substitute “horse” for “cow” on all lines except those that contained “black”

“The brown cow” -> “The brown horse”

“The black cow” -> “The black cow”

Print

The Print command (p) can be used to force the pattern space to be output, useful if the -n option has been specified

Syntax: [address1[,address2]]p

Note: if the -n option has not been specified, p will cause the line to be output twice!

Examples:

                                1,5p will display lines 1 through 5

                                /^$/,$p will display the lines from the first blank line through the last line of the file

Substitute

Syntax: [address(es)]s/pattern/replacement/[flags]

pattern - search pattern

replacement - replacement string for pattern

flags - optionally any of the following

n             a number from 1 to 512 indicating which               occurrence of pattern should be                                               replaced

g              global, replace all occurrences of pattern in pattern space

p             print contents of pattern space

 

Substitute Examples

The sed command for substitute patterns

s/Puff Daddy/P. Diddy/

Substitute P. Diddy for the first occurrence of Puff Daddy in pattern space

s/Tom/Dick/2

Substitutes Dick for the second occurrence of Tom in the pattern space

s/wood/plastic/p

Substitutes plastic for the first occurrence of wood and outputs (prints) pattern space

Video Tutorial: sed substitute http://www./watch?v=gSd42XDmKfU

Replacement Patterns

Substitute can use several special characters in the replacement string

& - replaced by the entire string matched in the regular expression for pattern

\n - replaced by the nth substring (or subexpression) previously specified using “\(“ and “\)”

\  -  used to escape the ampersand (&) and the backslash (\)

Replacement Pattern Examples

"the UNIX operating system …"

s/.NI./wonderful &/

"the wonderful UNIX operating system …"

Let us operate on the file with sed command

cat test1

first:second

one:two

sed 's/\(.*\):\(.*\)/\2:\1/' test1

second:first

two:one

sed 's/\([[:alpha:]]\)\([^ \n]*\)/\2\1ay/g'

Pig Latin ("unix is fun" -> "nixuay siay unfay")

Transform

The Transform command (y) operates like tr, it does a one-to-one or character-to-character replacement

Transform accepts zero, one or two addresses

[address[,address]]y/abc/xyz/

every a within the specified address(es) is transformed to an x.  The same is true for b to y and c to z

y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/ changes all lower case characters on the addressed line to upper case

If you only want to transform specific characters (or a word) in the line, it is much more difficult and requires use of the hold space

Quit

Quit causes sed to stop reading new input lines and stop sending them to standard output

It takes at most a single line address

Once a line matching the address is reached, the script will be terminated

This can be used to save time when you only want to process some portion of the beginning of a file

Example: to print the first 100 lines of a file (like head) use:

 

sed '100q' filename

 

sed will, by default, send the first 100 lines of filename to standard output and then quit processing

 

Sed Advantages

• We can use Regular expressions
• Soooper Fast
• it's code is so concise

 

Regular Expressions

^                                             matches the beginning of the line

$                                              matches the end of the line

.                                               Matches any single character

\                                              Escapes any metacharacter that follows, including itself.

(character)*                       Match arbitrarily many occurences of (character)

(character)?                       Match 0 or 1 instance of (character)

(character)+                       Match 1 or more instances of (character)

[abcdef]                               Match any character enclosed in [ ] (in this instance, a b c d e or f)

[^abcdef]                            Match any character NOT enclosed in [ ]

(character)\{m,n\}          Match m-n repetitions of (character)

(character)\{m,\}             Match m or more repetitions of (character)

(character)\{,n\}              Match n or less (possibly 0) repetitions of (character)

(character)\{n\}                Match exactly n repetitions of (character)

\{n,m\}                                 range of occurrences, n and m are integers

\(expression\)  Group operator.

expression1|expression2            Matches expression1 or expression 2.

()                                             groups regular expressions

Regular Expressions (character classes)

The following character classes are short-hand for matching special characters. 

                [:alnum:]             Printable characters (includes white space)

                [:alpha:]               Alphabetic characters

                [:blank:]               Space and tab characters

                [:cntrl:] Control characters

                [:digit:] Numeric characters        

                [:graph:]              Printable and visible (non-space) characters

                [:lower:]              Lowercase characters

                [:print:]                Alphanumeric characters

                [:punct:]              Punctuation characters

                [:space:]              Whitespace characters

                [:upper:]              Uppercase characters

                [:xdigit:]               Hexadecimal digits

Sed  Drawbacks

Hard to remember text from one line to another

Not possible to go backward in the file

No way to do forward references like    /..../+1

No facilities to manipulate numbers

Cumbersome syntax

 

SED reference: http://www./c/a/Server-Administration/Stream-Editor-in-the-UNIX-Shell/