NET-212 (NDG Linux Chapter 10)

10.1 Introduction

A large number of the files in a typical file system are text files. Text files only contain text, no formatting features that you might see in a word processing file. Because there are so many of these files on a typical Linux system, a significant number of commands exist to help users manipulate text files. There are commands to both view and modify these files in various ways. Additionally, there are features available for the shell to control the output of commands, so instead of having the output placed in the terminal window, the output can be redirected into another file or another command. These redirection features provide users with a much more flexible and powerful environment to work within.

[ ]

A list or range of characters to match one character If the first character within the brackets is the caret ^, it means any character not in the list

|

Alternation or like a logical "or" operator

10.3 Viewing Large Text Files

Although large text files can be viewed with the cat command, it is inconvenient to scroll backwards towards the top of the file. Additionally, really large files can't be displayed in this manner because the terminal window only stores a specific number of lines of output in memory. The use of the more or less commands allows for the user to the view data a "page" or a line at a time. These "pager" commands also permit other forms of navigation and searching that will be demonstrated in this section. Note Examples are given using both the more and less commands. Most of the time, the commands work the same, however, the less command is more advanced and has more features. The more command is still important to know because some Linux distributions don't have the less command, but all Linux distributions have the more command. If you are not interested in viewing the entire file or output of a command, then there are numerous commands that are able to filter the contents of the file or output. In this module, you will learn the use of the head and tail commands to be able to extract information from the top or bottom of the output of a command or file contents.

10.2.11 Step 11

Another popular form of redirection is to take the output of one command and send it into another command as input. For example, the output of some commands can be massive, resulting in the output scrolling off the screen too quickly to read. Execute the following command to take the output of the ls command and send it into the more command, which displays one page of data at a time: ls -l /etc | more Your output should be similar to the following: sysadmin@localhost:~$ ls -l /etc | more total 372 -rw-r--r-- 1 root root 2981 Jan 28 2015 adduser.conf -rw-r--r-- 1 root root 10 Jan 28 2015 adjtime drwxr-xr-x 1 root root 900 Jan 29 2015 alternatives drwxr-xr-x 1 root root 114 Jan 29 2015 apparmor.d drwxr-xr-x 1 root root 168 Oct 1 2014 apt -rw-r--r-- 1 root root 2076 Apr 3 2012 bash.bashrc drwxr-xr-x 1 root root 72 Jan 28 2015 bash_completion.d drwxr-sr-x 1 root bind 342 Jan 29 2015 bind -rw-r--r-- 1 root root 356 Apr 19 2012 bindresvport.blacklist -rw-r--r-- 1 root root 321 Mar 30 2012 blkid.conf lrwxrwxrwx 1 root root 15 Jun 18 2014 blkid.tab -> /dev/.blkid.tab drwxr-xr-x 1 root root 16 Jan 29 2015 ca-certificates -rw-r--r-- 1 root root 7464 Jan 29 2015 ca-certificates.conf drwxr-xr-x 1 root root 14 Jan 29 2015 calendar drwxr-xr-x 1 root root 24 Jan 29 2015 cron.d drwxr-xr-x 1 root root 134 Jan 29 2015 cron.daily drwxr-xr-x 1 root root 24 Jan 29 2015 cron.hourly drwxr-xr-x 1 root root 24 Jan 29 2015 cron.monthly -rw-r--r-- 1 root root 2969 Mar 15 2012 debconf.conf --More-- You will need to press the spacebar to continue or you can also press CTRL+c to escape this listing. The cut command is useful for extracting fields from files that are either delimited by a character, like the colon : in /etc/passwd, or that have a fixed width. It will be used in the next few examples as it typically provides a great deal of output that we can use to demonstrate using the | character.

10.3.10 Step 10

Another way to specify how many lines to output with the head command is to use the option -n -#, where # is the number of lines counted from the bottom of the output to exclude. Notice the minus symbol - in front of the #. For example, if the /etc/passwd contains 27 lines, the following command will display lines 1-7, excluding the last twenty lines: head -n -20 /etc/passwd sysadmin@localhost:~$ head -n -20 /etc/passwd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/sbin/nologin sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/usr/sbin/nologin man:x:6:12:man:/var/cache/man:/usr/sbin/nologin sysadmin@localhost:~$

.

Any single character

wc

Command that displays statistics of files such as the number of lines, words, and bytes.

head

Command used to display only the first few lines of a file.

tail

Command used to display only the last few lines of a files.

cat

Command used to display text files on standard output.

cut

Command used to extract columns of text from a file or standard input.

grep

Command used to filter lines in a file or the output of another command that matches a specified pattern.

sort

Command used to rearrange lines of files or input.

10.3.9 Step 9

Execute the following command line to pipe the output of the ls command to the tail command, displaying the last five file names in the /etc directory: ls /etc | tail -5 Your output should be similar to the following: sysadmin@localhost:~$ ls /etc | tail -5 updatedb.conf vim vtrgb wgetrc xdg sysadmin@localhost:~$ As you've seen, both head and tail commands output ten lines by default. You could also use an option -# (or you can use the option -n #, where # is a number of lines to output). Both commands can be used to read standard input from a pipe that receives output from a command. You have also seen where head and tail commands are different: the head command starts counting its lines to output from the top of the data, whereas the tail command counts the number of lines to output from the bottom of the data. There are some additional differences between these two commands as demonstrated in the next few tasks.

10.2.10 Step 10

Execute the following commands to use the tr command by redirecting stdin from a file: cat myfile tr a-z A-Z < myfile Your output should be similar to the following: sysadmin@localhost:~$ cat myfile wow, i see now this works! sysadmin@localhost:~$ tr a-z A-Z < myfile WOW, I SEE NOW THIS WORKS! sysadmin@localhost:~$

^

If the first character in the pattern, the pattern must be at the beginning of the line to match, otherwise just a literal ^ character

$

If the last character in the pattern, the pattern must be at the end of the line to match, otherwise just a literal $ character

10.8.5 The Backslash \ Character

In some cases, you may want to match a character that happens to be a special regular expression character. For example, consider the following: sysadmin@localhost:~/Documents$ cat newhome.txt Thanks for purchasing your new home!! **Warning** it may be haunted. There are three bathrooms. **Beware** of the ghost in the bedroom. The kitchen is open for entertaining. **Caution** the spirits don't like guests. Good luck!!! sysadmin@localhost:~/Documents$ grep 're*' newhome.txt Thanks for purchasing your new home!! **Warning** it may be haunted. There are three bathrooms. **Beware** of the ghost in the bedroom. The kitchen is open for entertaining. **Caution** the spirits don't like guests. In the output of the grep command above, the search for re* matched every line which contained an r followed by zero or more of the letter e. To look for an actual asterisk * character, place a backslash \ character before the asterisk * character: sysadmin@localhost:~/Documents$ grep 're\*' newhome.txt **Beware** of the ghost in the bedroom.

10.2.12 Step 12

In the following example, you will use a command called cut to extract all of the usernames from a database called /etc/passwd (a file that contains user account information). First, try running the cut command by itself: cut -d: -f1 /etc/passwd A portion of the command output is shown in the graphic below. sysadmin@localhost:~$ cut -d: -f1 /etc/passwd root daemon bin sys sync games man lp mail news uucp proxy www-data backup list irc gnats nobody libuuid syslog bind sshd operator

10.4 Searching Text Using Regular Expressions

In this task, you will use the grep family of commands with regular expressions in order to search for a specific string of characters in a stream of data (for example, a text file). The grep command uses basic regular expressions, special characters like wildcards that match patterns in data. The grep command returns the entire line containing the pattern that matches. The -E option to the grep command can be used to perform searches with extended regular expressions, essentially more powerful regular expressions. Another way to use extended regular expressions is to use the egrep command. The fgrep command is used to match literal characters, ignoring the special meaning of regular expression characters.

10.3 Input/Output Redirection

Input/Output (I/O) redirection allows for command line information to be passed to different streams. Before discussing redirection, it is important to understand the standard streams. STDIN Standard input, or STDIN, is information entered normally by the user via the keyboard. When a command prompts the shell for data, the shell provides the user with the ability to type commands that, in turn, are sent to the command as STDIN. STDOUT Standard output, or STDOUT, is the normal output of commands. When a command functions correctly (without errors) the output it produces is called STDOUT. By default, STDOUT is displayed in the terminal window where the command is executing. STDOUT is also known as stream or channel #1. STDERR Standard error, or STDERR, is error messages generated by commands. By default, STDERR is displayed in the terminal window where the command is executing. STDERR is also known as stream or channel #2. ‌⁠​​⁠​ I/O redirection allows the user to redirect STDIN so that data comes from a file and STDOUT/STDERR so that output goes to a file. Redirection is achieved by using the arrow < > characters.

Input/Output (I/O) redirection

Input/Output (I/O) redirection allows for command line information to be passed to different streams. Before discussing redirection, it is important to understand the standard streams. I/O redirection allows the user to redirect STDIN so that data comes from a file and STDOUT/STDERR so that output goes to a file. Redirection is achieved by using the arrow < > characters.

10.3.3 Redirecting Multiple Streams

It is possible to direct both the STDOUT and STDERR of a command at the same time. The following command produces both STDOUT and STDERR because one of the specified directories exists and the other does not: sysadmin@localhost:~$ ls /fake /etc/ppp ls: cannot access /fake: No such file or directory /etc/ppp: ip-down.d ip-up.d If only the STDOUT is sent to a file, STDERR is still printed to the screen: sysadmin@localhost:~$ ls /fake /etc/ppp > example.txt ls: cannot access /fake: No such file or directory sysadmin@localhost:~$ cat example.txt /etc/ppp: ip-down.d ip-up.d If only the STDERR is sent to a file, STDOUT is still printed to the screen: sysadmin@localhost:~$ ls /fake /etc/ppp 2> error.txt /etc/ppp: ip-down.d ip-up.d sysadmin@localhost:~$ cat error.txt ls: cannot access /fake: No such file or directory Both STDOUT and STDERR can be sent to a file by using the ampersand & character in front of the arrow > character. The &> character set means both 1> and 2>: sysadmin@localhost:~$ ls /fake /etc/ppp &> all.txt sysadmin@localhost:~$ cat all.txt ls: cannot access /fake: No such file or directory /etc/ppp: ip-down.d ip-up.d Note that when you use &>, the output appears in the file with all of the STDERR messages at the top and all of the STDOUT messages below all STDERR messages: sysadmin@localhost:~$ ls /fake /etc/ppp /junk /etc/sound &> all.txt sysadmin@localhost:~$ cat all.txt ls: cannot access '/fake': No such file or directory ls: cannot access '/junk': No such file or directory ls: cannot access '/etc/sound': No such file or directory /etc/ppp: ip-down.d ip-up.d If you don't want STDERR and STDOUT to both go to the same file, they can be redirected to different files by using both > and 2>. For example, to direct STDOUT to example.txt and STDERR to error.txt execute the following: sysadmin@localhost:~$ ls /fake /etc/ppp > example.txt 2> error.txt sysadmin@localhost:~$ cat error.txt ls: cannot access /fake: No such file or directory sysadmin@localhost:~$ cat example.txt /etc/ppp: ip-down.d ip-up.d The order in which the streams are specified does not matter.

10.4.4 Step 4

Match the pattern sync anywhere on a line: grep 'sync' passwd Your output should be similar to the following: sysadmin@localhost:/etc$ grep 'sync' passwd sync:x:4:65534:sync:/bin:/bin/sync sysadmin@localhost:/etc$

+

Matches previous character repeated one or more times

?

Matches previous character zero or one time, so it is an optional character

10.2 Command Line Pipes and Redirection

Normally, when you execute a command, the output is displayed in the terminal window. This output (also called a channel) is called standard output, symbolized by the term stdout. The file descriptor number for this channel is 1. Standard error (stderr) occurs when an error appears during the execution of a command; it has a file descriptor of 2. Error messages are also sent to the terminal window by default. In this lab, you will use characters that redirect the output from standard output (stdout) and standard error (stderr) to a file or to another command instead of the terminal screen. Standard input, stdin, usually is provided by you to a command by typing on the keyboard; it has a file descriptor of 0. However, by redirecting standard input, files can also be used as stdin.

10.2.14 Step 14

Now the output is sorted, but it still scrolls off the screen. Send the output of the sort command to the more command to solve this problem: cut -d: -f1 /etc/passwd | sort | more sysadmin@localhost:~$ cut -d: -f1 /etc/passwd | sort | more backup bin bind daemon games gnats irc libuuid list lp mail man news nobody operator proxy root sshd sync sys sysadmin syslog uucp --More--

-k1n

Numerically sort by field #1

10.8.1 The Period . Character

One of the most useful expressions is the period . character. It matches any character except for the new line character. Consider the unfiltered contents of the ~/Documents/red.txt file: sysadmin@localhost:~/Documents$ cat red.txt red reef rot reeed rd rod roof reed root reel read The pattern r..f would find any line that contained the letter r followed by exactly two characters and then the letter f: sysadmin@localhost:~/Documents$ grep 'r..f' red.txt reef roof The line does not have to be an exact match, it simply must contain the pattern, as seen here when r..t is searched for in the /etc/passwd file: sysadmin@localhost:~/Documents$ grep 'r..t' /etc/passwd root:x:0:0:root:/root:/bin/bash operator:x:1000:37::/root: The period character can be used any number of times. To find all words that have at least four characters, the following pattern can be used: sysadmin@localhost:~/Documents$ grep '....' red.txt reef reeed roof reed root reel read

less

Pager command used to view file contents one page of data at a time.

Pager commands

Pager commands display one page of data at a time, allowing you to move forward and backward in the file by using movement keys. For larger files, use a pager command to view the contents. There are two commonly used pager commands: The less command provides a very advanced paging capability. It is usually the default pager used by commands like the man command. The more command has been around since the early days of UNIX. While it has fewer features than the less command, however, the less command isn't included with all Linux distributions. The more command is always available.

10.3.4 Step 4

Press the Spacebar to view the rest of the document: <SPACE> In the next example, you will learn how to search a document using either the more or less commands. Searching for a pattern within both the more and less commands is done by typing the slash /, followed by the pattern to find. If a match is found, the screen should scroll to the first match. To move forward to the next match, press the n key. With the less command you can also move backwards to previous matches by pressing the N (capital n) key.

10.4.2 Step 2

Regular expressions are "greedy" in the sense that they will match every single instance of the specified pattern: grep root passwd Your output should be similar to the following: sysadmin@localhost:/etc$ grep root passwd root:x:0:0:root:/root:/bin/bash operator:x:1000:37::/root:/bin/sh sysadmin@localhost:/etc$ Note the red highlights indicate what exactly was matched. You can also see that all occurrences of root were matched on each line.

10.8 Basic Regular Expressions

Regular expressions, also referred to as regex, are a collection of normal and special characters that are used to find simple or complex patterns, respectively, in files. These characters are characters that are used to perform a particular matching function in a search. Normal characters are alphanumeric characters which match themselves. For example, an a would match an a. Special characters have special meanings when used within patterns by commands like the grep command. They behave in a more complex manner and do not match themselves. There are both Basic Regular Expressions (available to a wide variety of Linux commands) and Extended Regular Expressions (available to more advanced Linux commands). Basic Regular Expressions include the following: (On cards 34-38).

10.3.2 STDERR

STDERR can be redirected similarly to STDOUT. When using the arrow character to redirect, stream #1 (STDOUT) is assumed unless another stream is specified. Thus, stream #2 must be specified when redirecting STDERR by placing the number 2 preceding the arrow > character. To demonstrate redirecting STDERR, first observe the following command which produces an error because the specified directory does not exist: sysadmin@localhost:~$ ls /fake ls: cannot access /fake: No such file or directory Note that there is nothing in the example above that implies that the output is STDERR. The output is clearly an error message, but how could you tell that it is being sent to STDERR? One easy way to determine this is to redirect STDOUT: In the example above, STDOUT was redirected to the output.txt file. So, the output that is displayed can't be STDOUT because it would have been placed in the output.txt file instead of the terminal. Because all command output goes either to STDOUT or STDERR, the output displayed above must be STDERR. sysadmin@localhost:~$ ls /fake > output.txt ls: cannot access /fake: No such file or directory In the example above, STDOUT was redirected to the output.txt file. So, the output that is displayed can't be STDOUT because it would have been placed in the output.txt file instead of the terminal. Because all command output goes either to STDOUT or STDERR, the output displayed above must be STDERR. The STDERR output of a command can be sent to a file: sysadmin@localhost:~$ ls /fake 2> error.txt In the example, the 2> indicates that all error messages should be sent to the file error.txt, which can be confirmed using the cat command: sysadmin@localhost:~$ cat error.txt ls: cannot access /fake: No such file or directory

10.3.1 STDOUT

STDOUT can be directed to files. To begin, observe the output of the following echo command which displays to the screen: sysadmin@localhost:~$ echo "Line 1" Line 1 Using the > character, the output can be redirected to a file instead: sysadmin@localhost:~$ echo "Line 1" > example.txt This command displays no output because STDOUT was sent to the file example.txt instead of the screen. You can see the new file with the output of the ls command. sysadmin@localhost:~$ ls Desktop Downloads Pictures Templates example.txt Documents Music Public Videos The file contains the output of the echo command, which can be viewed with the cat command: sysadmin@localhost:~$ cat example.txt Line 1 It is important to realize that the single arrow overwrites any contents of an existing file: sysadmin@localhost:~$ cat example.txt Line 1 sysadmin@localhost:~$ echo "New line 1" > example.txt sysadmin@localhost:~$ cat example.txt New line 1 The original contents of the file are gone, replaced with the output of the new echo command. It is also possible to preserve the contents of an existing file by appending to it. Use two arrow >> characters to append to a file instead of overwriting it: sysadmin@localhost:~$ cat example.txt New line 1 sysadmin@localhost:~$ echo "Another line" >> example.txt sysadmin@localhost:~$ cat example.txt New line 1 Another line Instead of being overwritten, the output of the echo command is added to the bottom of the file.

-k2

Sort by field #2

-k3

Sort by field #3

-t,

Specifies the comma character as the field delimiter

Standard error or STDERR

Standard error, or STDERR, is error messages generated by commands. By default, STDERR is displayed in the terminal window where the command is executing. STDERR is also known as stream or channel #2.

10.2.8 Step 8

Standard input (stdin) can also be redirected. Normally stdin comes from the keyboard, but sometimes you want it to come from a file instead. For example, the tr command translates characters, but it only accepts data from stdin, never from a file name given as an argument. This is great when you want to do something like capitalize data that is inputted from the keyboard (Note: Press Control+d, to signal the tr command to stop processing standard input): tr a-z A-Z this is interesting how do I stop this? ^D Your output should be similar to the following: sysadmin@localhost:~$ tr a-z A-Z this is interesting THIS IS INTERESTING how do I stop this? HOW DO I STOP THIS? sysadmin@localhost:~$ Note: ^D symbolizes Control+d

Standard input or STDIN

Standard input, or STDIN, is information entered normally by the user via the keyboard. When a command prompts the shell for data, the shell provides the user with the ability to type commands that, in turn, are sent to the command as STDIN.

Standard output or STDOUT

Standard output, or STDOUT, is the normal output of commands. When a command functions correctly (without errors) the output it produces is called STDOUT. By default, STDOUT is displayed in the terminal window where the command is executing. STDOUT is also known as stream or channel #1.

10.4.11 Step 11

Suppose you want to search for a pattern containing a sequence of three digits. You can use { } characters with a number to express that you want to repeat a pattern a specific number of times; for example: {3}. The use of the numeric qualifier requires the extended mode of grep: grep -E '[0-9]{3}' passwd Your output should be similar to the following: sysadmin@localhost:/etc$ grep -E '[0-9]{3}' passwd sync:x:4:65534:sync:/bin:/bin/sync nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin _apt:x:100:65534::/nonexistent:/usr/sbin/nologin systemd-network:x:101:102:systemd Network Management,,,:/run/systemd/netif:/usr/ sbin/nologin systemd-resolve:x:102:103:systemd Resolver,,,:/run/systemd/resolve:/usr/sbin/nol ogin syslog:x:103:106::/home/syslog:/usr/sbin/nologin messagebus:x:104:107::/nonexistent:/usr/sbin/nologin bind:x:105:110::/var/cache/bind:/usr/sbin/nologin sshd:x:106:65534::/run/sshd:/usr/sbin/nologin operator:x:1000:37::/root:/bin/sh sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash sysadmin@localhost:/etc$

-k3

The -k option specifies the field number. To specify which field to sort by, use the -k option with an argument to indicate the field number, starting with 1 for the first field. The following example uses the -k3 option to sort by the third field.

-t:

The -t option specifies the field delimiter. If the file or input is separated by a delimiter other than whitespace, for example a comma or colon, the -t option will allow for another field separator to be specified as an argument. The mypasswd file used in the previous example uses a colon : character as a delimiter to separate the fields, so the following example uses the -t: option.

10.3.1 Step 1

The /etc/passwd is likely too large to be displayed on the screen without scrolling the screen. To see a demonstration of this, use the cat command to display the entire contents of the /etc/passwdfile: cat /etc/passwd Your output should be similar to the following: sysadmin@localhost:~$ cat /etc/passwd daemon:x:1:1:daemon:/usr/sbin:/bin/sh bin:x:2:2:bin:/bin:/bin/sh sys:x:3:3:sys:/dev:/bin/sh sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/bin/sh man:x:6:12:man:/var/cache/man:/bin/sh lp:x:7:7:lp:/var/spool/lpd:/bin/sh mail:x:8:8:mail:/var/mail:/bin/sh news:x:9:9:news:/var/spool/news:/bin/sh uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh proxy:x:13:13:proxy:/bin:/bin/sh www-data:x:33:33:www-data:/var/www:/bin/sh backup:x:34:34:backup:/var/backups:/bin/sh list:x:38:38:Mailing List Manager:/var/list:/bin/sh irc:x:39:39:ircd:/var/run/ircd:/bin/sh gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh nobody:x:65534:65534:nobody:/nonexistent:/bin/sh libuuid:x:100:101::/var/lib/libuuid:/bin/sh syslog:x:101:103::/home/syslog:/bin/false bind:x:102:105::/var/cache/bind:/bin/false sshd:x:103:65534::/var/run/sshd:/usr/sbin/nologin operator:x:1000:37::/root:/bin/sh sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash sysadmin@localhost:~$

10.4.10 Step 10

The [ ] characters can also be used to match a single character. However, unlike the period character ., the [ ] characters are used to specify exactly what character you want to match. For example, if you want to match a numeric character, you can specify [0-9]. Execute the following command for a demonstration: head passwd | grep '[0-9]' Your output should be similar to the following: sysadmin@localhost:/etc$ head passwd | grep '[0-9]' root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/sbin/nologin sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/usr/sbin/nologin man:x:6:12:man:/var/cache/man:/usr/sbin/nologin lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin mail:x:8:8:mail:/var/mail:/usr/sbin/nologin news:x:9:9:news:/var/spool/news:/usr/sbin/nologin sysadmin@localhost:/etc$ Note The head command was used to limit the output of the grep command.

10.8.3 The Asterisk * Character

The asterisk * character is used to match zero or more occurrences of a character or pattern preceding it. For example, e* would match zero or more occurrences of the letter e: sysadmin@localhost:~/Documents$ cat red.txt red reef rot reeed rd rod roof reed root reel read sysadmin@localhost:~/Documents$ grep 're*d' red.txt red reeed rd reed It is also possible to match zero or more occurrences of a list of characters by utilizing the square brackets. The pattern [oe]* used in the following example matches zero or more occurrences of the o character or the e character: sysadmin@localhost:~/Documents$ grep 'r[oe]*d' red.txt red reeed rd rod reed When used with only one other character, * isn't very helpful. Any of the following patterns would match every string or line in the file: '.*' 'e*' 'b*' 'z*' because the asterisk * character can match zero occurrences of a pattern. sysadmin@localhost:~/Documents$ grep 'z*' red.txt red reef rot reeed rd rod roof reed root reel read sysadmin@localhost:~/Documents$ grep 'e*' red.txt red reef rot reeed rd rod roof reed root reel read To make the asterisk character useful, it is necessary to create a pattern which includes more than just the one character preceding it. For example, the results above can be refined by adding another e to make the pattern ee* effectively matching every line which contains at least one e. sysadmin@localhost:~/Documents$ grep 'ee*' red.txt red reef reeed reed reel read

cat command

The cat command, short for concatenate, is a simple but useful command whose functions include creating and displaying text files, as well as combining copies of text files. One of the most popular uses of cat is to display the content of text files.

10.1.1 Viewing Files in the Terminal

The cat command, short for concatenate, is a simple but useful command whose functions include creating and displaying text files, as well as combining copies of text files. One of the most popular uses of cat is to display the content of text files. To display a file in the standard output using the cat command, type the command followed by the filename: sysadmin@localhost:~$ cd Documents sysadmin@localhost:~/Documents$ cat food.txt Food is good. Although the terminal is the default output of this command, the cat command can also be used for redirecting file content to other files or input for another command by using redirection characters.

10.3.4 STDIN

The concept of redirecting STDIN is a difficult one because it is more difficult to understand why you would want to redirect STDIN. With STDOUT and STDERR, their purpose is straightforward; sometimes it is helpful to store the output into a file for future use. Most Linux users end up redirecting STDOUT routinely, STDERR on occasion, and STDIN very rarely. There are very few commands that require you to redirect STDIN because with most commands if you want to read data from a file into a command, you can specify the filename as an argument to the command. For some commands, if you don't specify a filename as an argument, they revert to using STDIN to get data. For example, consider the following cat command: sysadmin@localhost:~$ cat hello hello‌⁠​​⁠​ how are you? how are you? goodbye goodbye Note If you do attempt the cat command without arguments, kill the process and return to the prompt by using Ctrl+C. In the preceding example, the cat command isn't provided a filename as an argument. So, it asks for the data to display on the screen from STDIN. The user types hello, and then the cat command displays hello on the screen. While this is mildly entertaining, it isn't particularly useful. However, if the output of the cat command were redirected to a file, then this method could be used either to add text to an existing file or to place text into a new file. The first command in the example below redirects the output of the cat command to a newly created file called new.txt. This action is followed up by providing the cat command with the new.txt file as an argument to display the redirected text in STDOUT. sysadmin@localhost:~$ cat > new.txt Hello How are you? Goodbye sysadmin@localhost:~$ cat new.txt Hello How are you? Goodbye While the previous example demonstrates another advantage of redirecting STDOUT, it doesn't address why or how STDIN can be directed. To understand this, consider a new command called tr. This command takes a set of characters and translates them into another set of characters. For example, to capitalize a line of text use the tr command as follows: sysadmin@localhost:~$ tr 'a-z' 'A-Z' watch how this works WATCH HOW THIS WORKS The tr command took the STDIN from the keyboard and converted all lower-case letters before sending STDOUT to the screen. It would seem that a better use of the tr command would be to perform translation on a file, not keyboard input. However, the tr command does not support file name arguments: sysadmin@localhost:~$ cat example.txt /etc/ppp: ip-down.d ip-up.d sysadmin@localhost:~$ tr 'a-z' 'A-Z' example.txt tr: extra operand `example.txt' Try `tr --help' for more information It is possible, however, to tell the shell to get STDIN from a file instead of from the keyboard by using the < character: sysadmin@localhost:~$ tr 'a-z' 'A-Z' < example.txt /ETC/PPP: IP-DOWN.D IP-UP.D Most commands do accept file names as arguments, so this use case is relatively rare. However, for those that do not, this method could be used to have the shell read from the file instead of relying on the command to have this ability. One last note to save the resulting output, redirect it into another file: sysadmin@localhost:~$ tr 'a-z' 'A-Z' < example.txt > newexample.txt sysadmin@localhost:~$ cat newexample.txt /ETC/PPP: IP-DOWN.D IP-UP.D Most commands do accept file names as arguments, so this use case is relatively rare. However, for those that do not, this method could be used to have the shell read from the file instead of relying on the command to have this ability. One last note to save the resulting output, redirect it into another file: sysadmin@localhost:~$ tr 'a-z' 'A-Z' < example.txt > newexample.txt sysadmin@localhost:~$ cat newexample.txt /ETC/PPP: IP-DOWN.D IP-UP.D

10.6 Filter File Sections

The cut command can extract columns of text from a file or standard input. It's primarily used for working with delimited database files. Again, delimited files are files that contain columns separated by a delimiter. These files are very common on Linux systems. By default, the cut command expects its input to be separated by the tab character, but the -d option can specify alternative delimiters such as the colon or comma. The -f option can specify which fields to display, either as a hyphenated range or a comma-separated list. In the following example, the first, fifth, sixth and seventh fields from the mypasswd database file are displayed: sysadmin@localhost:~$ cut -d: -f1,5-7 mypasswd root:root:/root:/bin/bash daemon:daemon:/usr/sbin:/usr/sbin/nologin bin:bin:/bin:/usr/sbin/nologin sys:sys:/dev:/usr/sbin/nologin sync:sync:/bin:/bin/sync The cut command is also able to extract columns of text based upon character position with the -c option—useful when working with fixed-width database files or command outputs. For example, the fields of the ls -l command are always in the same character positions. The following will display just the file type (character 1), permissions (characters 2-10), a space (character 11), and filename (characters 50+): sysadmin@localhost:~$ ls -l | cut -c1-11,50- total 44 drwxr-xr-x Desktop drwxr-xr-x Documents drwxr-xr-x Downloads drwxr-xr-x Music drwxr-xr-x Pictures drwxr-xr-x Public drwxr-xr-x Templates drwxr-xr-x Videos -rw-rw-r-- all.txt -rw-rw-r-- example.txt -rw-rw-r-- mypasswd -rw-rw-r-- new.txt

10.2.4 Step 4

The find command is a good command to demonstrate how stderr works. This very flexible command allows searching with a host of options such as filename, size, date, type and permission. The find command will begin the search in the directory specified and recursively search all of the subdirectories. For example, to search for files beginning in your home directory containing the name bash: find ~ -name "*bash*" Your output should be similar to the following: sysadmin@localhost:~$ find ~ -name "*bash*" /home/sysadmin/.bash_logout /home/sysadmin/.bashrc sysadmin@localhost:~$ Remember that ~ is used to represent your home directory. Now, run the following command and observe the output: find /etc -name hosts Your output should be similar to the following: sysadmin@localhost:~$ find /etc -name hosts /etc/hosts find: '/etc/ssl/private': Permission denied sysadmin@localhost:~$ Notice the error message indicating you do not have permission to access certain files/directories. This is because as a regular user, you don't have the right to "look inside" some directories. These types of error messages are sent to stderr, not stdout. The find command is beyond the scope of this course. The purpose of using the command is to demonstrate the difference between stdout and stderr.

10.7 Filter File Contents

The grep command can be used to filter lines in a file or the output of another command that matches a specified pattern. That pattern can be as simple as the exact text that you want to match or it can be much more advanced through the use of regular expressions. For example, to find all the users who can log in to the system with the BASH shell, the grep command can be used to filter the lines from the /etc/passwd file for the lines containing the pattern bash: sysadmin@localhost:~$ grep bash /etc/passwd root:x:0:0:root:/root:/bin/bash sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash To make it easier to see what exactly is matched, use the --color option. This option will highlight the matched items in red: sysadmin@localhost:~$ grep --color bash /etc/passwd root:x:0:0:root:/root:/bin/bash sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash Note On our virtual machines, the grep command is aliased to include the --color option automatically. In some cases, it may not be important to find the specific lines that match the pattern, but rather how many lines match the pattern. The -c option provides a count of how many lines match: sysadmin@localhost:~$ grep -c bash /etc/passwd 2 When viewing the output from the grep command, it can be hard to determine the original line numbers. This information can be useful when going back into the file (perhaps to edit the file) to quickly find one of the matched lines. The -n option to the grep command will display original line numbers. To display all lines and their line numbers in the /etc/passwd file which contain the pattern bash: sysadmin@localhost:~$ grep -n bash /etc/passwd 1:root:x:0:0:root:/root:/bin/bash 27:sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash The -v option inverts the match, outputting all lines that do not contain the pattern. To display all lines not containing nologin in the /etc/passwd file: sysadmin@localhost:~$ grep -v nologin /etc/passwd root:x:0:0:root:/root:/bin/bash sync:x:4:65534:sync:/bin:/bin/sync operator:x:1000:37::/root:/bin/sh sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash The -i option ignores the case (capitalization) distinctions. The following searches for the pattern the in newhome.txt, allowing each character to be uppercase or lowercase: sysadmin@localhost:~$ cd Documents sysadmin@localhost:~/Documents$ grep -i the newhome.txt There are three bathrooms. **Beware** of the ghost in the bedroom. The kitchen is open for entertaining. **Caution** the spirits don't like guests. The -w option only returns lines which contain matches that form whole words. To be a word, the character string must be preceded and followed by a non-word character. Word characters include letters, digits, and the underscore character. The following examples search for the are pattern in the newhome.txt file. The first command searches with no options, while the second command includes the -w option. Compare the outputs: sysadmin@localhost:~/Documents$ grep are newhome.txt There are three bathrooms. **Beware** of the ghost in the bedroom. sysadmin@localhost:~/Documents$ grep -w are newhome.txt There are three bathrooms.

10.1.3 Head and Tail

The head and tail commands are used to display only the first few or last few lines of a file, respectively (or, when used with a pipe, the output of a previous command). By default, the head and tail commands display ten lines of the file that is provided as an argument. For example, the following command displays the first ten lines of the /etc/sysctl.conf file: sysadmin@localhost:~/Documents$ cd sysadmin@localhost:~$ head /etc/sysctl.conf # # /etc/sysctl.conf - Configuration file for setting system variables # See /etc/sysctl.d/ for additional system variables # See sysctl.conf (5) for information. # #kernel.domainname = example.com # Uncomment the following to stop low-level messages on console #kernel.printk = 3 4 1 3 Passing a number as an option will cause both the head and tail commands to output the specified number of lines, instead of the standard ten. For example to display the last five lines of the /etc/sysctl.conf file use the -5 option: sysadmin@localhost:~$ tail -5 /etc/sysctl.conf # Protects against creating or following links under certain conditions # Debian kernels have both set to 1 (restricted) # See https://www.kernel.org/doc/Documentation/sysctl/fs.txt #fs.protected_hardlinks=0 #fs.protected_symlinks=0 The -n option can also be used to indicate how many lines to output. Pass a number as an argument to the option: sysadmin@localhost:~$ head -n 3 /etc/sysctl.conf # # /etc/sysctl.conf - Configuration file for setting system variables # See /etc/sysctl.d/ for additional system variables Negative Value Option Traditionally in UNIX, the number of lines to output would be specified as an option with either command, so -3 meant to show three lines. For the tail command, either -3 or -n -3 still means show three lines. However, the GNU version of the head command recognizes -n -3 as show all but the last three lines, and yet the head command still recognizes the option -3 as show the first three lines. Positive Value Option The GNU version of the tail command allows for a variation of how to specify the number of lines to be printed. If the -n option is used with a number prefixed by the plus sign, then the tail command recognizes this to mean to display the contents starting at the specified line and continuing all the way to the end. For example, the following displays the contents of the /etc/passwd from line 25 to the end of the file: sysadmin@localhost:~$ nl /etc/passwd | tail -n +25 25 sshd:x:103:65534::/var/run/sshd:/usr/sbin/nologin 26 operator:x:1000:37::/root:/bin/sh 27 sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash Consider This Live file changes can be viewed by using the -f option to the tail command—useful when you want to see changes to a file as they are happening. A good example of this would be when viewing log files as a system administrator. Log files can be used to troubleshoot problems and administrators often view them "interactively" with the tail command while performing commands in a separate window. For example, if you were to log in as the root user, you could troubleshoot issues with the email server by viewing live changes to the /var/log/mail.log log file.

10.2.13 Step 13

The output in the previous example was unordered and scrolled off the screen. In the next step you are going to take the output of the cut command and send it into the sort command to provide some order to the output: cut -d: -f1 /etc/passwd | sort A portion of the command output is shown in the graphic below. sysadmin@localhost:~$ cut -d: -f1 /etc/passwd | sort backup bin bind daemon games gnats irc libuuid list lp mail man news nobody operator proxy root sshd sync sys

10.4.7 Step 7

The pipe character, |, or "alternation operator", acts as an "or" operator. For example, execute the following to attempt to match either sshd, root or operator: grep 'sshd|root|operator' passwd Your output should be similar to the following: sysadmin@localhost:/etc$ grep 'sshd|root|operator' passwd sysadmin@localhost:/etc$ Observe that the grep command does not recognize the pipe as the alternation operator by default. The grep command is actually including the pipe as a plain character in the pattern to be matched. The use of either grep -E or egrep will allow the use of the extended regular expressions, including alternation.

10.2 Command Line Pipes

The pipe | character can be used to send the output of one command to another. Typically, when a command has output or generates an error, the output is displayed to the screen; however, this does not have to be the case. Instead of being printed to the screen, the output of one command becomes input for the next command. This tool can be powerful, especially when looking for specific data; piping is often used to refine the results of an initial command. In previous examples the head and tail commands were given files as arguments to operate on. However, the pipe character allows you to utilize these commands not only on files, but on the output of other commands. This can be useful when listing a large directory, for example the /etc directory: sysadmin@localhost:~$ ls /etc X11 gss mke2fs.conf rpc adduser.conf host.conf modprobe.d rsyslog.conf alternatives hostname modules rsyslog.d apparmor hosts modules-load.d securetty apparmor.d hosts.allow motd security apt hosts.deny mtab selinux bash.bashrc init.d nanorc services bind initramfs-tools netplan shadow bindresvport.blacklist inputrc network shadow- binfmt.d insserv.conf.d networks shells ca-certificates iproute2 newt skel ca-certificates.conf issue nsswitch.conf ssh calendar issue.net opt ssl console-setup kernel os-release subgid cron.d ld.so.cache pam.conf subgid- cron.daily ld.so.conf pam.d subuid cron.hourly ld.so.conf.d passwd subuid- cron.monthly ldap passwd- sudoers cron.weekly legal perl sudoers.d crontab libaudit.conf pinforc sysctl.conf dbus-1 locale.alias ppp sysctl.d debconf.conf locale.gen profile systemd debian_version localtime profile.d terminfo default logcheck protocols timezone deluser.conf login.defs python3 tmpfiles.d depmod.d logrotate.conf python3.6 ucf.conf dhcp logrotate.d rc0.d udev dpkg lsb-release rc1.d ufw environment machine-id rc2.d update-motd.d fstab magic rc3.d updatedb.conf gai.conf magic.mime rc4.d vim groff mailcap rc5.d vtrgb group mailcap.order rc6.d wgetrc group- manpath.config rcS.d xdg gshadow mc resolv.conf gshadow- mime.types rmt The previous command lists a large number of files. If you execute this in our terminal, the output is cut off and can only be viewed if scrolling up. To more easily view the beginning of the output, pipe it to the head command. The following example displays only the first ten lines: sysadmin@localhost:~$ ls /etc | head X11 adduser.conf alternatives apparmor apparmor.d apt bash.bashrc bind bindresvport.blacklist binfmt.d The full output of the ls command is passed to the head command by the shell instead of being printed to the screen. The head command takes this output from the ls command as input data, and the output of head is then printed to the screen. Multiple pipes can be used consecutively to link multiple commands together. If three commands are piped together, the output of the first command is passed to the second command. Then, the output of the second command is passed to the third command. The output of the third command would then be printed to the screen. It is important to carefully choose the order in which commands are piped, as each command only sees input from the previous command. The examples below illustrate this using the nl command, which adds line numbers to the output. In the first example, the nl command is used to number the lines of the output of the preceding ls command: sysadmin@localhost:~$ ls /etc/ssh | nl 1 moduli 2 ssh_config 3 ssh_host_ecdsa_key 4 ssh_host_ecdsa_key.pub 5 ssh_host_ed25519_key 6 ssh_host_ed25519_key.pub 7 ssh_host_rsa_key 8 ssh_host_rsa_key.pub 9 ssh_import_id 10 sshd_config In the next example, note that the ls command is executed first and its output is sent to the nl command, numbering all of the lines from the output of the ls command. Then the tail command is executed, displaying the last five lines from the output of the nl command: sysadmin@localhost:~$ ls /etc/ssh | nl | tail -5 6 ssh_host_ed25519_key.pub 7 ssh_host_rsa_key 8 ssh_host_rsa_key.pub 9 ssh_import_id 10 sshd_config Compare the output above with the next example: sysadmin@localhost:~$ ls /etc/ssh | tail -5 | nl 1 ssh_host_ed25519_key.pub 2 ssh_host_rsa_key 3 ssh_host_rsa_key.pub 4 ssh_import_id 5 sshd_config Notice how the line numbers are different. Why is this? In the second example, the output of the ls command is first sent to the tail command which only grabs the last five lines of the output. Then the tail command sends those five lines to the nl command, which numbers them 1-5. Pipes can be powerful, but it is necessary to consider how commands are piped to ensure that the desired output is displayed.

*

The previous character repeated zero or more times

10.4 Sorting Files or Input

The sort command can be used to rearrange the lines of files or input in either dictionary or numeric order. The following example creates a small file, using the head command to grab the first 5 lines of the /etc/passwd file and send the output to a file called mypasswd. sysadmin@localhost:~$ head -5 /etc/passwd > mypasswd sysadmin@localhost:~$ cat mypasswd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/sbin/nologin sync:x:4:65534:sync:/bin:/bin/sync Now we will sort the mypasswd file: sysadmin@localhost:~$ sort mypasswd bin:x:2:2:bin:/bin:/usr/sbin/nologin daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin root:x:0:0:root:/root:/bin/bash sync:x:4:65534:sync:/bin:/bin/sync sys:x:3:3:sys:/dev:/usr/sbin/nologin Upon close examination of the output in the preceding example, the sort command has arranged the lines of the file in alphabetical order. Compare this output to the output of the previous cat command.

10.4.1 Fields and Sort Options

The sort command can rearrange the output based on the contents of one or more fields. Fields are determined by a field delimiter contained on each line. In computing, a delimiter is a character that separates a string of text or data; it defaults to whitespace, like spaces or tabs. The following command can be used to sort the third field of the mypasswd file numerically. Three options are used to achieve this sort: (On cards 21-23).

10.2.9 Step 9

The tr command accepts keyboard input (stdin), translates the characters and then redirects the output to stdout. To create a file of all lower-case characters, execute the following: tr A-Z a-z > myfile Wow, I SEE NOW This WORKS! Your output should be similar to the following: sysadmin@localhost:~$ tr A-Z a-z > myfile Wow, I SEE NOW This WORKS! sysadmin@localhost:~$ Press the Enter key to make sure your cursor is on the line below "This works!", then use Control+d to stop input. To verify you created the file, execute the following command: cat myfile Your output should be similar to the following: sysadmin@localhost:~$ cat myfile wow, i see now this works! sysadmin@localhost:~$

10.8.6 Extended Regular Expressions

The use of extended regular expressions often requires a special option be provided to the command to recognize them. Historically, there is a command called egrep, which is similar to grep, but can understand extended regular expressions. Now, the egrep command is deprecated in favor of using grep with the -E option. The following regular expressions are considered extended: (On cards 46-48).

10.4.1 Step 1

The use of grep in its simplest form is to search for a given string of characters, such as sshd in the /etc/passwd file. The grep command will print the entire line containing the match: cd /etc grep sshd passwd Your output should be similar to the following: sysadmin@localhost:~$ cd /etc sysadmin@localhost:/etc$ grep sshd passwd sshd:x:106:65534::/run/sshd:/usr/sbin/nologin sysadmin@localhost:/etc$

10.5 Viewing File Statistics

The wc command provides the number of lines, words and bytes (1 byte = 1 character in a text file) for a file, and a total line count if more than one file is specified. By default, the wc command allows for up to three statistics to be printed for each file provided, as well as the total of these statistics if more than one filename is provided: sysadmin@localhost:~$ wc /etc/passwd /etc/passwd- 35 56 1710 /etc/passwd 34 55 1665 /etc/passwd- 69 111 3375 total The output of the previous example has four columns: 1. Number of lines 2. Number of words 3. Number of bytes 4. File name It is also possible to view only specific statistics, by using the -l option to show just the number of lines, the -w option to show just the number of words, the -c option to show just the number of bytes, or any combination of these options. The wc command can be useful for counting the number of lines output by some other command through a pipe. For example, if you wanted to know the total number of files in the /etc directory, pipe the output of ls to wc and count only the number of lines: sysadmin@localhost:~$ ls /etc/ | wc -l 142

10.1.2.2 Pager Searching Commands

There are two ways to search in the less command: searching forward or backward from your current position. To start a search to look forward from your current position, use the slash / key. Then, type the text or pattern to match and press the Enter key. Abdul Abdul's Abe /frog If a match can be found, then the cursor moves in the document to the match. For example, in the following graphic the expression "frog" was searched for in the words file: bullfrog bullfrog's bullfrogs bullheaded bullhorn bullhorn's Notice that "frog" didn't have to be a word by itself. Also notice that while the less command moved to the first match from the current position, all matches were highlighted. If no matches forward from your current position can be found, then the last line of the screen will report Pattern not found: Pattern not found (press RETURN) To search backward from your current position, press the question mark ? key, then type the text or pattern to match and press the Enter key. The cursor moves backward to the first match it can find or reports that the pattern cannot be found. If more than one match can be found by a search, then use the n key to move the next match and use the Shift+N key combination to go to a previous match. The search terms actually use patterns called regular expressions. More details regarding regular expressions are provided later in this chapter.

10.1 Introduction

This is Lab 10: Working with Text. By performing this lab, students will learn how to redirect text streams, use regular expressions and use commands for filtering text files. In this lab, you will perform the following tasks: Learn how to redirect and pipe standard input, output and error channels. Use regular expressions to filter the output of commands or file content. View large files or command output with programs for paging, and viewing selected portions.

-n

This option specifies the sort type. The third field in the mypasswd file contains numbers, so the -n option is used to perform a numeric sort.

10.4.3 Step 3

To limit the output, you can use regular expressions to specify a more precise pattern. For example, the caret ^ character can be used to match a pattern at the beginning of a line; so, when you execute the following command line, only lines that begin with root should be matched and displayed: grep '^root' passwd Your output should be similar to the following: sysadmin@localhost:/etc$ grep '^root' passwd root:x:0:0:root:/root:/bin/bash sysadmin@localhost:/etc$ Note that there are two additional instances of the word root but only the one appearing at the beginning of the line is matched (displayed in red). Best Practice Use single quotes (not double quotes) around regular expressions to prevent the shell program from trying to interpret them.

10.2.7 Step 7

To redirect both standard output (stdout) and standard error (stderr) to one file, first redirect stdout to a file and then redirect stderr to that same file by using the notation 2>&1. find /etc -name hosts > find.out 2>&1 cat find.out Your output should be similar to the following: sysadmin@localhost:~$ find /etc -name hosts > find.out 2>&1 sysadmin@localhost:~$ cat find.out /etc/hosts find: '/etc/ssl/private': Permission denied sysadmin@localhost:~$ The 2>&1 part of the command means "send the stderr (channel 2) to the same place where stdout (channel 1) is going".

10.2.5 Step 5

To redirect stderr (error messages) to a file, issue the following command: find /etc -name hosts 2> err.txt cat err.txt Your output should be similar to the following: sysadmin@localhost:~$ find /etc -name hosts 2> err.txt /etc/hosts sysadmin@localhost:~$ cat err.txt find: `/etc/ssl/private': Permission denied sysadmin@localhost:~$ Recall that the file descriptor for stderr is the number 2, so it is used along with the > symbol to redirect the stderr output to a file called err.txt. Note that 1> is the same as >. The previous example demonstrates why knowing redirection is important. If you want to "ignore" the errors that the find command displays, you can redirect those messages into a file and look at them later, making it easier to focus on the rest of the output of the command.

10.1.2.1 Pager Movement Commands

To view a file with the less command, pass the file name as an argument: sysadmin@localhost:~/Documents$ less words There are many movement commands for the less command, each with multiple possible keys or key combinations. While this may seem intimidating, it is not necessary to memorize all of these movement commands. When viewing a file with the less command, use the H key or Shift+H to display a help screen: SUMMARY OF LESS COMMANDS Commands marked with * may be preceded by a number, N. Notes in parentheses indicate the behavior if N is given. A key preceded by a caret indicates the Ctrl key; thus ^K is ctrl-K. h H Display this help. q :q Q :Q ZZ Exit. ------------------------------------------------------------------------ MOVING e ^E j ^N CR * Forward one line (or N lines). y ^Y k ^K ^P * Backward one line (or N lines). f ^F ^V SPACE * Forward one window (or N lines). b ^B ESC-v * Backward one window (or N lines). z * Forward one window (and set window to N). w * Backward one window (and set window to N). ESC-SPACE * Forward one window, but don't stop at end-of-file. d ^D * Forward one half-window (and set half-window to N). u ^U * Backward one half-window (and set half-window to N). ESC-) RightArrow * Left one half screen width (or N positions). ESC-( LeftArrow * Right one half screen width (or N positions). HELP -- Press RETURN for more, or q when done The first group of movement commands to focus on are the ones that are most commonly used. To make it even more convenient, the keys that are identical in more and less are summarized below in order to demonstrate how to move in more and less at the same time: Key Movement Spacebar Window forward B Window backward Enter Line forward Q Exit H Help When using less as a pager, the easiest way to advance forward a page is to press the Spacebar.

10.4.9 Step 9

Use another extended regular expression, this time with egrep with alternation in a group to match a pattern. The strings nob and non will match: egrep 'no(b|n)' passwd Your output should be similar to the following: sysadmin@localhost:/etc$ egrep 'no(b|n)' passwd nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin _apt:x:100:65534::/nonexistent:/usr/sbin/nologin messagebus:x:104:107::/nonexistent:/usr/sbin/nologin sysadmin@localhost:/etc$ Note The parentheses, ( ), were used to limit the "scope" of the | character. Without them, a pattern such as nob|n would have meant "match either nob or n".

10.4.5 Step 5

Use the $ symbol to match the pattern sync at the end of a line: grep 'sync$' passwd Your output should be similar to the following: sysadmin@localhost:/etc$ grep 'sync$' passwd sync:x:4:65534:sync:/bin:/bin/sync sysadmin@localhost:/etc$ The command in the previous step matched every instance; the second only matches the instance at the end of the line.

10.4.8 Step 8

Use the -E switch to allow grep to operate in extended mode in order to recognize the alternation operator: grep -E 'sshd|root|operator' passwd Your output should be similar to the following: sysadmin@localhost:/etc$ grep -E 'sshd|root|operator' passwd root:x:0:0:root:/root:/bin/bash sshd:x:103:65534::/var/run/sshd:/usr/sbin/nologin operator:x:1000:37::/root:/bin/sh sysadmin@localhost:/etc$

10.3.8 Step 8

Use the head command to display the first two lines of the /etc/passwd file: head -2 /etc/passwd Your output should be similar to the following: sysadmin@localhost:~$ head -2 /etc/passwd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin sysadmin@localhost:~$

10.3.5 Step 5

Use the less command to display the entire contents of the /etc/passwd file. Then search for the word bin, use n to move forward, and N to move backwards. Finally, quit the less pager by typing the letter q: less /etc/passwd /bin nnnNNNq Important Unlike the more command which automatically exits when you reach the end of a file, you must press a quit key such as q to quit the less program.

10.3.2 Step 2

Use the more command to display the entire contents of the /etc/passwd file: more /etc/passwd Your output should be similar to the following: sysadmin@localhost:~$ more /etc/passwd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/bin/sh bin:x:2:2:bin:/bin:/bin/sh sys:x:3:3:sys:/dev:/bin/sh sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/bin/sh man:x:6:12:man:/var/cache/man:/bin/sh lp:x:7:7:lp:/var/spool/lpd:/bin/sh mail:x:8:8:mail:/var/mail:/bin/sh news:x:9:9:news:/var/spool/news:/bin/sh uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh proxy:x:13:13:proxy:/bin:/bin/sh www-data:x:33:33:www-data:/var/www:/bin/sh backup:x:34:34:backup:/var/backups:/bin/sh list:x:38:38:Mailing List Manager:/var/list:/bin/sh irc:x:39:39:ircd:/var/run/ircd:/bin/sh gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh nobody:x:65534:65534:nobody:/nonexistent:/bin/sh libuuid:x:100:101::/var/lib/libuuid:/bin/sh syslog:x:101:103::/home/syslog:/bin/false bind:x:102:105::/var/cache/bind:/bin/false sshd:x:103:65534::/var/run/sshd:/usr/sbin/nologin operator:x:1000:37::/root:/bin/sh --More--(92%) Note The --More--(92%) indicates you are "in" the more command and 92% through the current data.

10.4.6 Step 6

Use the period character . to match any single character. For example, execute the following command to match any character followed by a 'y': grep '.y' passwd Your output should be similar to the following: sysadmin@localhost:/etc$ grep '.y' passwd sys:x:3:3:sys:/dev:/usr/sbin/nologin sync:x:4:65534:sync:/bin:/bin/sync proxy:x:13:13:proxy:/bin:/usr/sbin/nologin gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin systemd-network:x:101:102:systemd Network Management,,,:/run/systemd/netif:/usr/ sbin/nologin systemd-resolve:x:102:103:systemd Resolver,,,:/run/systemd/resolve:/usr/sbin/nologin syslog:x:103:106::/home/syslog:/usr/sbin/nologin sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash sysadmin@localhost:/etc$

10.2.1 Step 1

Use the redirection symbol > along with the echo command to redirect the output from the normal output of stdout (to the terminal) to a file. The cat command can be used to display file contents and will be used in this example to verify redirected output to the file. Type the following: echo "Hello World" echo "Hello World" > mymessage cat mymessage Your output should be similar to the following: sysadmin@localhost:~$ echo "Hello World" Hello World sysadmin@localhost:~$ echo "Hello World" > mymessage sysadmin@localhost:~$ cat mymessage Hello World sysadmin@localhost:~$ The first command echoes the message (stdout) to the terminal. The second command redirects the output; instead of sending the output to the terminal, it is sent to a file called mymessage. The last command displays the contents of the mymessage file.

10.3.7 Step 7

Use the tail command to display the last ten lines of the /etc/passwd file: tail /etc/passwd Your output should be similar to the following: sysadmin@localhost:~$ tail /etc/passwd nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin _apt:x:100:65534::/nonexistent:/usr/sbin/nologin systemd-network:x:101:102:systemd Network Management,,,:/run/systemd/netif:/usr/ sbin/nologin systemd-resolve:x:102:103:systemd Resolver,,,:/run/systemd/resolve:/usr/sbin/nol ogin syslog:x:103:106::/home/syslog:/usr/sbin/nologin messagebus:x:104:107::/nonexistent:/usr/sbin/nologin bind:x:105:110::/var/cache/bind:/usr/sbin/nologin sshd:x:106:65534::/run/sshd:/usr/sbin/nologin operator:x:1000:37::/root:/bin/sh sysadmin:x:1001:1001:System Administrator,,,,:/home/sysadmin:/bin/bash sysadmin@localhost:~$

10.8.4 Anchor Characters

When performing a pattern match, the match could occur anywhere on the line. Anchor characters are one of the ways regular expressions can be used to narrow down search results. They specify whether the match occurs at the beginning of the line or the end of the line. For example, the pattern root appears many times in the /etc/passwd file: sysadmin@localhost:~/Documents$ grep 'root' /etc/passwd root:x:0:0:root:/root:/bin/bash operator:x:1000:37::/root: The caret (circumflex) ^ character is used to ensure that a pattern appears at the beginning of the line. For example, to find all lines in /etc/passwd that start with root use the pattern ^root. Note that ^ must be the first character in the pattern to be effective: sysadmin@localhost:~/Documents$ grep '^root' /etc/passwd root:x:0:0:root:/root:/bin/bash The second anchor character $ can be used to ensure a pattern appears at the end of the line, thereby effectively reducing the search results. To find the lines that end with an r in the alpha-first.txt file, use the pattern r$: sysadmin@localhost:~/Documents$ cat alpha-first.txt A is for Animal B is for Bear C is for Cat D is for Dog E is for Elephant F is for Flower sysadmin@localhost:~/Documents$ grep 'r$' alpha-first.txt B is for Bear F is for Flower Again, the position of this character is important. The $ must be the last character in the pattern to be effective as an anchor.

10.8.2 The Bracket [ ] Characters

When using the . character, any possible character could match it. In some cases, you want to specify exactly which characters you want to match, such as a lowercase alphabet character or a number character. The square brackets [ ] match a single character from the list or range of possible characters contained within the brackets. For example, given the profile.txt file: sysadmin@localhost:~/Documents$ cat profile.txt Hello my name is Joe. I am 37 years old. 3121991 My favorite food is avocados. I have 2 dogs. 123456789101112 To find all the lines in profile.txt which have a number in them, use the pattern [0123456789] or [0-9]: sysadmin@localhost:~/Documents$ grep '[0-9]' profile.txt I am 37 years old. 3121991 I have 2 dogs. 123456789101112 Note that each possible character can be listed out [abcd] or provided as a range [a-d], as long as the range is in the correct order. For example, [d-a] wouldn't work because it isn't a valid range: sysadmin@localhost:~/Documents$ grep '[d-a]' profile.txt grep: Invalid range end The range is specified by a standard called the ASCII table. This table is a collection of all printable characters in a specific order. You can see the ASCII table with the ascii command. A small sample: 041 33 21 ! 141 97 61 a 042 34 22 " 142 98 62 b 043 35 23 # 143 99 63 c 044 36 24 $ 144 100 64 d 045 37 25 % 145 101 65 e 046 38 26 & 146 102 66 f The ASCII value of the letter a is 97 while the value of d is 100. Since 97 is smaller than 100, the range a-d (97-100) is a valid range. What about exempting characters?, For instance, to match a character that can be anything except an x, y or z? It would be inefficient to provide a set with all of the characters except x, y or z. To match a character that is not one of the listed characters, start the set with a ^ symbol. To find all the lines which contain any non-numeric characters, insert a ^ as the first character inside the brackets. This character negates the characters listed: sysadmin@localhost:~/Documents$ grep '[^0-9]' profile.txt Hello my name is Joe. I am 37 years old. My favorite food is avocados. I have 2 dogs. Consider This Do not mistake [^0-9] to match lines which do not contain numbers. It actually matches lines which contain non-numbers. Look at the original file to see the difference. The third and sixth lines only contain numbers; they do not contain non-numbers, so those lines do not match.

10.2.2 Step 2

When you use the > symbol to redirect stdout, the contents of the file are first destroyed. Type the following commands to see a demonstration: echo "Greetings" > mymessage cat mymessage Your output should be similar to the following: sysadmin@localhost:~$ echo "Greetings" > mymessage sysadmin@localhost:~$ cat mymessage Greetings sysadmin@localhost:~$ Notice that using one redirection symbol overwrites an existing file. This is called "clobbering" a file.

10.1.2 Viewing Files Using a Pager

While viewing small files with the cat command poses no problems, it is not an ideal choice for large files. The cat command doesn't provide any easy ways to pause and restart the display, so the entire file contents are dumped to the screen. For larger files, use a pager command to view the contents. Pager commands display one page of data at a time, allowing you to move forward and backward in the file by using movement keys. There are two commonly used pager commands: The less command provides a very advanced paging capability. It is usually the default pager used by commands like the man command. The more command has been around since the early days of UNIX. While it has fewer features than the less command, however, the less command isn't included with all Linux distributions. The more command is always available. The more and less commands allow users to move around the document using keystroke commands. Because developers based the less command on the functionality of the more command, all of the keystroke commands available in the more command also work in the less command. The focus of our content is on the more advanced less command. The more command is still useful to remember for times when the less command isn't available. Remember that most of the keystroke commands provided work for both commands.

10.3.3 Step 3

While you are in the more command, you can view the help screen by pressing the h key: h Your output should be similar to the following: Most commands optionally preceded by integer argument k. Defaults in brackets Star (*) indicates argument becomes new default. ------------------------------------------------------------------------------ <space> Display next k lines of text [current screen size] z Display next k lines of text [current screen size]* <return> Display next k lines of text [1]* d or ctrl-D Scroll k lines [current scroll size, initially 11]* q or Q or <interrupt> Exit from more s Skip forward k lines of text [1] f Skip forward k screenfuls of text [1] b or ctrl-B Skip backwards k screenfuls of text [1] ' Go to place where previous search started = Display current line number /<regular expression> Search for kth occurrence of regular expression [1] n Search for kth occurrence of last r.e [1] !<cmd> or :!<cmd> Execute <cmd> in a subshell v Start up /usr/bin/vi at current line ctrl-L Redraw screen :n Go to kth next file [1] :p Go to kth previous file [1] :f Display current file name and line number . Repeat previous command ------------------------------------------------------------------------------ --More--(92%)

10.2.6 Step 6

You can also redirect stdout and stderr into two separate files. find /etc -name hosts > std.out 2> std.err cat std.err cat std.out Your output should be similar to the following: sysadmin@localhost:~$ find /etc -name hosts > std.out 2> std.err sysadmin@localhost:~$ cat std.err find: `/etc/ssl/private': Permission denied sysadmin@localhost:~$ cat std.out /etc/hosts Note that a space is permitted but not required after the > redirection symbol.

10.2.3 Step 3

You can avoid clobbering a file by using >> instead of >. By using >> you append to a file. Execute the following commands to see a demonstration of this: cat mymessage echo "How are you?" >> mymessage cat mymessage Your output should be similar to the following: sysadmin@localhost:~$ cat mymessage Greetings sysadmin@localhost:~$ echo "How are you?" >> mymessage sysadmin@localhost:~$ cat mymessage Greetings How are you? sysadmin@localhost:~$ Notice that by using >> all existing data is preserved and the new data is appended at the end of the file.

10.3.6 Step 6

You can use the head command to display the top part of a file. By default, the head command will display the first ten lines of the file: head /etc/passwd Your output should be similar to the following: sysadmin@localhost:~$ head /etc/passwd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/bin/sh bin:x:2:2:bin:/bin:/bin/sh sys:x:3:3:sys:/dev:/bin/sh sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/bin/sh man:x:6:12:man:/var/cache/man:/bin/sh lp:x:7:7:lp:/var/spool/lpd:/bin/sh mail:x:8:8:mail:/var/mail:/bin/sh news:x:9:9:news:/var/spool/news:/bin/sh sysadmin@localhost:~$