grep, sed, awk

sed command add line, change line, transform

#To add a line after a match, use the a command sed '/pattern/ a "Add this line"' file ex. sed '/unix/ a "Add a new line" file.txt #Prints "Add a new line" after each line containing unix #To add a line before a math, use the i command sed '/pattern/ i "Add this line"' file ex. sed '/unix/ i "Add a new line" file.txt #Prints "Add a new line" before each line containing unix #To change a line, the c command replaces the enter line with the new line sed '/pattern/ c "Change to this line"' file ex. sed '/unix/ c "Meow"' file.txt #Replaces any line containing unix with "Meow" #To transform the text, can convert lowercase letters to uppercase using the y command sed 'y/lowercaseletters/UPPERCASELETTERS' file ex. sed 'y/ul/UL' file.txt #Converts each lowercase u or l to an uppercase u or l

sed command deletion and duplication

Deletion: Specify the line number or range of numbers and use the 'd' character sed 'n d' file #Deletes the n line of the file ex. sed '2 d' file.txt #Deletes the second line ex. sed '2,$ d' file.txt #Deletes the second through last lines of the file Duplication: Specify a number, range of number, or no numbers of line(s) to duplicate and use the 'p' character sed 'p' file #Prints each line twice ex. sed '2 p' file #Prints the second line twice

awk built-in variables

FS: Regex used to separate fields FILENAME: Name of the current input file OFS: Output field separator RS: Input record separator ORS: Output record separator NF: Number of fields on current line NR: Current record number

awk

Interpreted programming language which focuses on processing text Searches files for text containing a pattern, and when a line or text matches, performs a specific action on that line or text Syntax for an awk program: 'BEGIN {start_action} {action} END {stop_action}' file #The actions in the BEGIN block are performed before processing, and the actions in the END block are performed after processing http://www.folkstalk.com/2011/12/good-examples-of-awk-command-in-unix.html https://www.tutorialspoint.com/unix_commands/awk.htm

egrep command

Searches for a pattern using extended regular expressions Same as grep -E Same options as grep Syntax: egrep [options] PATTERN file

fgrep

Searches for fixed character strings in a file or files; fixed character means the string is interpreted literally, so metacharacters do not exist, and so regex cant be used This means it is useful to search for strings that may contain lots of metacharacters Syntax: fgrep [options] pattern file Same as running grep with -F option

sed command

Short for stream editor, allows you to modify files or replace text Syntax: sed [options] script inputfile

sed command substitution

Substitution: sed 's/wordtoreplace/newWord/' file #By default, replaces the first occurrence of the pattern in each line & wont replace the second, third,..etc #To replace multiple occurrences in a line, use /n flag where n is the occurrence to be replaced; the flag /g replaces all occurrences in a line; can combine /3g so all occurrences after 3 (including 3) are replaced #Can use a different delimiter than '/'; such as sed 's_wordtoreplace_newWord' file #To add to a matched string, the & stands for the matched string: sed 's/wordtoreplace/{&} file would add a { before the wordtoreplace and a } after the last character of the wordtoreplace #sed 's_\(wordtoreplace\)_\1\1_' file The first pair of parentheses represents \1, and the second pair represents \2, so ?? #sed 's/wordtoreplace/newWord/p' file Prints the replaced line twice #sed -n 's/wordtoreplace/newWord/p' file Prints only the replaced lines #Can pipe the output of a sed command into another sed command: sed 's/unix/linux/' file.txt | sed 's/os/system/' #Run multiple sed commands with -e option sed -e 's/unix/linux/' -e 's/os/system/ file.txt #To replace a string only on a specific line number, place a number N before substituion; n,n specifies a range of lines; sed '3 s/unix/linux/' file.txt #Replaces third line only #To replace on lines which match a pattern, place the /pattern/ before substitution sed '/linux/ s/unix/centos/' file.txt #Finds line containing linux, then replaces the word unix with centos

grep

Used to search the text or searches the given file for lines containing a match to the given strings or words By default, will display the matching lines Syntax: grep 'word' filename grep 'world' file1 file2 file3 grep 'string1 string2' filename cat otherfile | grep 'something' command | grep 'something' command option1 | grep 'something'

other awk functions

asort(arr) #Sorts the array using their contents asorti(arr) #Sorts the array using their indices

awk arguments

awk [ -F fs ] [ -v var=value ] [ 'prog' OR -f progfile ] [ file ... ] #Scans each input file for lines that match any of the specified patterns in prog or one or more input files specified by -f progfile #For each matching pattern, associated action is performed; if no action provided, means print the line; an action can be a choice statement, loop, statement, expression, etc -F fs #Sets the input field separator to the regular expression fs. -v var=value #Assigns the value value to the variable var before executing the awk program. 'prog' #An awk program. -f progfile #Specify a file, progfile, which contains the awk program to be executed. file ... #A file to be processed by the specified awk program.

fgrep options

fgrep -c #Print only the count of the lines that contain the pattern fgrep -i #Case insensitive fgrep -e pattern_list #Search only for a string in pattern_list fgrep -f pattern_file #Take the list of patterns from a pattern file

grep command options

grep -c #Instead print a count of matching lines for each input file grep -d ACTION #If an input file is a directory, use ACTION to process it; by default, action is read, which means directories are read like ordinary files; if ACTION is skip, skips the directory; if ACTION is recurse, grep reads all files under each directory recursively grep -e PATTERN #Use PATTERN as the pattern grep -E #Interpret the pattern as an extended regular expression grep -F #Interpret the pattern as a list of fixed strings, separated by newlines, any of which is to be matched grep -f file #Obtain patterns from the file grep -G PATTERN #Interpret the pattern as a basic regular expression grep -H #Print the filename for each match grep -i #Ignore case distinctions in both the pattern and file grep -l #list file name whose contents match grep -m NUM #Stop reading a file after NUM matching lines grep -n #Prefix each line of output with the line number grep -o #Show only the part of a matching line that matches pattern grep -r OR grep -R #Read all files under each directory recursively grep -w #Select only those lines containing matches that form whole words grep -x #Select only those matches that match the whole line ex. grep -i "boo" /etc/passwd #match boo, BOO, and all other variations ex. grep -w "boo" file #Only selects the lines containing the full word boo ex. grep -l 'main' *.c #Lists the file name of the .c files whose contents contain main ex. grep 'ia$' /usr/share/dict/words #Basic regular expression, where the pattern matches 'ia' characters at the end of a line

ADD MORE ON SED

http://www.computerhope.com/unix/used.htm

awk string functions

index(string,sub) #Checks if sub is a substring of string or not; returns position where sub starts or returns 0 if not a substring length(string) #Returns length of a string sub(regex,sub,string) #Replaces first occurence of sub in the string string with regex substr(string,position) #Returns substring of string from position to the end of the string substr(string,position,max) #Returns a substring of string, starting at the position and ending at max tolower(string) toupper(string) split(string, array, delimiter) #splits a string into an array using the delimiter match(str,regex) #Returns index of first longest match of regex in str, or 0 ex. awk '{ split($2,arr,","); if(arr[3] == "UNIX") print $0 } ' file.txt #Would print 1 U,N,UNIX,000 3 I,M,UNIX,222

Regular Expressions or Patterns

letters/numbers as they are . represents any one character ** is any string of characters ^ is the beginning of a line $ is the end of a line [characterList] #Matches any single character in the list; if the first character is ^, then it matches any character NOT in the list [char-char] #Matches the specified range of characters \w #Matches any number or letter (case insensitive) \W #Does not match with any number or letter (case insensitive) \< #Match the empty string at the beginning of a word \> #Match the empty string at the end of a word ? #Preceeding item is optional and matched at most once + #Preceeding item matched one or more times {n} #Preceeding item matched exactly n times {n,} #Preceeding item matched n or more times {n,m} #Preceeding item matched at least n times, but no more than m times subExpr|subExpr #Matches any string matching either subexpression ex. grep a.f #Looks for letters a & f separated by one character ex. grep a*f #Looks for a string beginning with a and ending with f (could be af, auf, auddjf); put this pattern inside single quotes