grep, sed, awk
sed command add line, change line, transform
#To add a line after a match, use the a command sed '/pattern/ a "Add this line"' file ex. sed '/unix/ a "Add a new line" file.txt #Prints "Add a new line" after each line containing unix #To add a line before a math, use the i command sed '/pattern/ i "Add this line"' file ex. sed '/unix/ i "Add a new line" file.txt #Prints "Add a new line" before each line containing unix #To change a line, the c command replaces the enter line with the new line sed '/pattern/ c "Change to this line"' file ex. sed '/unix/ c "Meow"' file.txt #Replaces any line containing unix with "Meow" #To transform the text, can convert lowercase letters to uppercase using the y command sed 'y/lowercaseletters/UPPERCASELETTERS' file ex. sed 'y/ul/UL' file.txt #Converts each lowercase u or l to an uppercase u or l
sed command deletion and duplication
Deletion: Specify the line number or range of numbers and use the 'd' character sed 'n d' file #Deletes the n line of the file ex. sed '2 d' file.txt #Deletes the second line ex. sed '2,$ d' file.txt #Deletes the second through last lines of the file Duplication: Specify a number, range of number, or no numbers of line(s) to duplicate and use the 'p' character sed 'p' file #Prints each line twice ex. sed '2 p' file #Prints the second line twice
awk built-in variables
FS: Regex used to separate fields FILENAME: Name of the current input file OFS: Output field separator RS: Input record separator ORS: Output record separator NF: Number of fields on current line NR: Current record number
awk
Interpreted programming language which focuses on processing text Searches files for text containing a pattern, and when a line or text matches, performs a specific action on that line or text Syntax for an awk program: 'BEGIN {start_action} {action} END {stop_action}' file #The actions in the BEGIN block are performed before processing, and the actions in the END block are performed after processing http://www.folkstalk.com/2011/12/good-examples-of-awk-command-in-unix.html https://www.tutorialspoint.com/unix_commands/awk.htm
egrep command
Searches for a pattern using extended regular expressions Same as grep -E Same options as grep Syntax: egrep [options] PATTERN file
fgrep
Searches for fixed character strings in a file or files; fixed character means the string is interpreted literally, so metacharacters do not exist, and so regex cant be used This means it is useful to search for strings that may contain lots of metacharacters Syntax: fgrep [options] pattern file Same as running grep with -F option
sed command
Short for stream editor, allows you to modify files or replace text Syntax: sed [options] script inputfile
sed command substitution
Substitution: sed 's/wordtoreplace/newWord/' file #By default, replaces the first occurrence of the pattern in each line & wont replace the second, third,..etc #To replace multiple occurrences in a line, use /n flag where n is the occurrence to be replaced; the flag /g replaces all occurrences in a line; can combine /3g so all occurrences after 3 (including 3) are replaced #Can use a different delimiter than '/'; such as sed 's_wordtoreplace_newWord' file #To add to a matched string, the & stands for the matched string: sed 's/wordtoreplace/{&} file would add a { before the wordtoreplace and a } after the last character of the wordtoreplace #sed 's_\(wordtoreplace\)_\1\1_' file The first pair of parentheses represents \1, and the second pair represents \2, so ?? #sed 's/wordtoreplace/newWord/p' file Prints the replaced line twice #sed -n 's/wordtoreplace/newWord/p' file Prints only the replaced lines #Can pipe the output of a sed command into another sed command: sed 's/unix/linux/' file.txt | sed 's/os/system/' #Run multiple sed commands with -e option sed -e 's/unix/linux/' -e 's/os/system/ file.txt #To replace a string only on a specific line number, place a number N before substituion; n,n specifies a range of lines; sed '3 s/unix/linux/' file.txt #Replaces third line only #To replace on lines which match a pattern, place the /pattern/ before substitution sed '/linux/ s/unix/centos/' file.txt #Finds line containing linux, then replaces the word unix with centos
grep
Used to search the text or searches the given file for lines containing a match to the given strings or words By default, will display the matching lines Syntax: grep 'word' filename grep 'world' file1 file2 file3 grep 'string1 string2' filename cat otherfile | grep 'something' command | grep 'something' command option1 | grep 'something'
other awk functions
asort(arr) #Sorts the array using their contents asorti(arr) #Sorts the array using their indices
awk arguments
awk [ -F fs ] [ -v var=value ] [ 'prog' OR -f progfile ] [ file ... ] #Scans each input file for lines that match any of the specified patterns in prog or one or more input files specified by -f progfile #For each matching pattern, associated action is performed; if no action provided, means print the line; an action can be a choice statement, loop, statement, expression, etc -F fs #Sets the input field separator to the regular expression fs. -v var=value #Assigns the value value to the variable var before executing the awk program. 'prog' #An awk program. -f progfile #Specify a file, progfile, which contains the awk program to be executed. file ... #A file to be processed by the specified awk program.
fgrep options
fgrep -c #Print only the count of the lines that contain the pattern fgrep -i #Case insensitive fgrep -e pattern_list #Search only for a string in pattern_list fgrep -f pattern_file #Take the list of patterns from a pattern file
grep command options
grep -c #Instead print a count of matching lines for each input file grep -d ACTION #If an input file is a directory, use ACTION to process it; by default, action is read, which means directories are read like ordinary files; if ACTION is skip, skips the directory; if ACTION is recurse, grep reads all files under each directory recursively grep -e PATTERN #Use PATTERN as the pattern grep -E #Interpret the pattern as an extended regular expression grep -F #Interpret the pattern as a list of fixed strings, separated by newlines, any of which is to be matched grep -f file #Obtain patterns from the file grep -G PATTERN #Interpret the pattern as a basic regular expression grep -H #Print the filename for each match grep -i #Ignore case distinctions in both the pattern and file grep -l #list file name whose contents match grep -m NUM #Stop reading a file after NUM matching lines grep -n #Prefix each line of output with the line number grep -o #Show only the part of a matching line that matches pattern grep -r OR grep -R #Read all files under each directory recursively grep -w #Select only those lines containing matches that form whole words grep -x #Select only those matches that match the whole line ex. grep -i "boo" /etc/passwd #match boo, BOO, and all other variations ex. grep -w "boo" file #Only selects the lines containing the full word boo ex. grep -l 'main' *.c #Lists the file name of the .c files whose contents contain main ex. grep 'ia$' /usr/share/dict/words #Basic regular expression, where the pattern matches 'ia' characters at the end of a line
ADD MORE ON SED
http://www.computerhope.com/unix/used.htm
awk string functions
index(string,sub) #Checks if sub is a substring of string or not; returns position where sub starts or returns 0 if not a substring length(string) #Returns length of a string sub(regex,sub,string) #Replaces first occurence of sub in the string string with regex substr(string,position) #Returns substring of string from position to the end of the string substr(string,position,max) #Returns a substring of string, starting at the position and ending at max tolower(string) toupper(string) split(string, array, delimiter) #splits a string into an array using the delimiter match(str,regex) #Returns index of first longest match of regex in str, or 0 ex. awk '{ split($2,arr,","); if(arr[3] == "UNIX") print $0 } ' file.txt #Would print 1 U,N,UNIX,000 3 I,M,UNIX,222
Regular Expressions or Patterns
letters/numbers as they are . represents any one character ** is any string of characters ^ is the beginning of a line $ is the end of a line [characterList] #Matches any single character in the list; if the first character is ^, then it matches any character NOT in the list [char-char] #Matches the specified range of characters \w #Matches any number or letter (case insensitive) \W #Does not match with any number or letter (case insensitive) \< #Match the empty string at the beginning of a word \> #Match the empty string at the end of a word ? #Preceeding item is optional and matched at most once + #Preceeding item matched one or more times {n} #Preceeding item matched exactly n times {n,} #Preceeding item matched n or more times {n,m} #Preceeding item matched at least n times, but no more than m times subExpr|subExpr #Matches any string matching either subexpression ex. grep a.f #Looks for letters a & f separated by one character ex. grep a*f #Looks for a string beginning with a and ending with f (could be af, auf, auddjf); put this pattern inside single quotes