Chapter 12
seven-step troubleshooting process
1. Define Problem 2. Gather Information 3. Propose Hypothesis 4. Analyze Information 5. Eliminate Possible Causes 4. /6. Test Hypothesis 5. / 7. Solve the Problem and Document Solution
Steps to Establish Network Baseline
1. Determine What Type of Data to Collect. start by selecting a few variables that represent the defined policies. Good starting variables are interface utilization and CPU utilization. 2. Identify Devices and Ports of Interest. Identify devices and ports which performance data should be measured. Network device ports that connect to other network devices Servers Key users Anything else considered critical to operations 3. Determine the Baseline Duration length of time and the baseline information being gathered must be long enough to determine a "normal" picture of the network. It is important that daily trends of network traffic are monitored. It is also important to monitor for trends that occur over a longer period, such as weekly or monthly. For this reason, when capturing data for analysis, the period specified should be, at a minimum, seven days long. baseline needs to last no more than six weeks, unless specific long-term trends need to be measured. Generally, a two-to-four-week baseline is adequate.
Step 2 - Check for Duplex Mismatches
Duplex configuration guidelines include the following: Autonegotiation of speed and duplex is recommended. If autonegotiation fails, manually set the speed and duplex on interconnecting ends. Point-to-point Ethernet links should always run in full-duplex mode. Half-duplex is uncommon and typically encountered only when legacy hubs are used. Duplex configuration guidelines include the following: Autonegotiation of speed and duplex is recommended. If autonegotiation fails, manually set the speed and duplex on interconnecting ends. Point-to-point Ethernet links should always run in full-duplex mode. Half-duplex is uncommon and typically encountered only when legacy hubs are used.
Top-down model
End-user applications of an end system are tested before tackling the more specific networking pieces. requires checking every network application until the possible cause of the problem is found. Start at Application Transport Network Data Link Physical
Network baseline should answer the following questions:
How does the network perform during a normal or average day? Where are the most errors occurring? What part of the network is most heavily used? What part of the network is least used? Which devices should be monitored and what alert thresholds should be set? Can the network meet the identified policies?
Step 4 - Verify Default Gateway
If there is no detailed route on the router, or if the host is configured with the wrong default gateway, then communication between two endpoints in different networks does not work. Use #show ip route | include Gateway|0.0.0.0 (on router) used to verify the default gateway. Use C:\> route print (on pc) used to verify the presence of the IPv4 default gateway as shown in the command output.
Show Interfaces Output
Input Queue Drops (and the related ignored and throttle counters) signify that at some point, more traffic was delivered to the router than it could process. This does not necessarily indicate a problem. That could be normal traffic during peak periods. However, it could be an indication that the CPU cannot process packets in time, so if this number is consistently high, it is worth trying to spot at which moments these counters are increasing and how this relates to CPU usage. Output Queue Drops indicate that packets were dropped due to congestion on the interface. Seeing output drops is normal for any point where the aggregate input traffic is higher than the output traffic. During peak traffic periods, packets are dropped if traffic is delivered to the interface faster than it can be sent out. However, even if this is considered normal behavior, it leads to packet drops and queuing delays, so applications that are sensitive to those, such as VoIP, might suffer from performance issues. Consistently seeing output queue drops can be an indicator that you need to implement an advanced queuing mechanism to implement or modify QoS. Input Errors indicate errors that are experienced during the reception of the frame, such as CRC errors. High numbers of CRC errors could indicate cabling problems, interface hardware problems, or, in an Ethernet-based network, duplex mismatches. Output Errors indicate errors, such as collisions, during the transmission of a frame. In most Ethernet-based networks today, full-duplex transmission is the norm, and half-duplex transmission is the exception. In full-duplex transmission, operation collisions cannot occur; therefore, collisions (especially late collisions) often indicate duplex mismatches.
Troubleshooting with Layered Models
Notice that routers and multilayer switches are shown at Layer 4, the transport layer. Although routers and multilayer switches usually make forwarding decisions at Layer 3, ACLs on these devices can be used to make filtering decisions using Layer 4 information. End System Application Presentation Session Router / Multilayer Switch Transport Network Standard Switch Data Link Hub Physical
types of network topology diagrams:
Physical Logical IPv4 Logical IPv6
Transport Layer Troubleshooting ACLs
Problems arise at the edge of the network where traffic is examined and modified. both access control lists (ACLs) and Network Address Translation (NAT) operate at the network layer and may involve operations at the transport layer, as shown in the figure. Symptoms Connectivity issues Access Issues Causes ACL configurations NAT configurations Common ACL Misconfigurations Selection of traffic flow Order of ACL entries Implicit deny any Address and IPv4 wildcard masks Selection of transport layer protocol Source and destination ports Use of the established keyword Uncommon protocols
Logical IPv6
See logical IPv4
Verify DNS
Show running-config
logic flowchart of a simplified three-stage troubleshooting process
Step 1: Gather Symptoms Step 2: Isolate the Problem Step 3: Implement Corrective Action Problem Fixed? Yes Document solution & save changes No If it did not fix the problem or if it created another problem, undo corrective action and start again.
Telnet output
Telnet w/ port # #telnet 2001:db8:acad:2::2 80
Troubleshooting End-to-End Connectivity
Two of the most common utilities used to verify a problem with end-to-end connectivity are ping and traceroute Step 1. Check physical connectivity at the point where network communication stops. This includes cables and hardware. The problem might be with a faulty cable or interface, or involve misconfigured or faulty hardware. Step 2. Check for duplex mismatches. Step 3. Check data link and network layer addressing on the local network. This includes IPv4 ARP tables, IPv6 neighbor tables, MAC address tables, and VLAN assignments. Step 4. Verify that the default gateway is correct. Step 5. Ensure that devices are determining the correct path from the source to the destination. Manipulate the routing information if necessary. Step 6. Verify the transport layer is functioning properly. Telnet can also be used to test transport layer connections from the command line. Step 7. Verify that there are no ACLs blocking traffic. Step 8. Ensure that DNS settings are correct. There should be a DNS server that is accessible. The outcome of this process is operational, end-to-end connectivity. If all the steps have been performed without any resolution, the network administrator may either want to repeat the previous steps or escalate the problem to a senior administrator.
Windows IPv6 Neighbor Table (netsh interface ipv6 show neighbor output)
Windows command output lists all devices that are currently in the neighbor table. The information that is displayed for each device includes the IPv6 address, physical (MAC) address, and the type of addressing. By examining the neighbor table, the network administrator can verify that destination IPv6 addresses map to correct Ethernet addresses. The IPv6 link-local addresses on all interfaces of R1 have been manually configured to FE80::1. Similarly, R2 has been configured with the link-local address of FE80::2 on its interfaces and R3 has been configured with the link-local address of FE80::3 on its interfaces. Remember, link-local addresses must be unique on the link or network.
Network Layer Troubleshooting
any problem that involves a Layer 3 protocol, such as IPv4, IPv6, EIGRP, OSPF, etc. Symptoms Network failure Suboptimal performance Causes General network issues Connectivity issues Routing table Neighbor issues Topology database
Windows ARP Table (arp -a output)
arp Windows command lists all devices that are currently in the ARP cache. The information that is displayed for each device includes the IPv4 address, physical (MAC) address, and the type of addressing (static or dynamic). The cache can be cleared by using the arp -d Windows command if the network administrator wants to repopulate the cache with updated information. Should show the default gateway
IPv4 and IPv6 routing tables
can be populated by the following methods: Directly connected networks Local host or local routes Static routes Dynamic routes Default routes The routing table process will attempt to forward the packet using an entry in the routing table with the greatest number of leftmost matching bits. The number of matching bits is indicated by the prefix length of the route.
Divide and Conquer
collecting user experiences of the problem, document the symptoms and then, using that information, make an informed guess as to which OSI layer to start your investigation. When a layer is verified to be functioning properly, it can be assumed that the layers below it are functioning. The administrator can work up the OSI layers. If an OSI layer is not functioning properly, the administrator can work down the OSI layer model.
Data Link Troubleshooting
configuration and operation of these protocols are critical to creating a functional, well-tuned network. Layer 2 problems cause specific symptoms that, when recognized, will help identify the problem quickly. Symptoms No functionality or connectivity at network layer or above Network operating below baseline performance levels Excessive broadcasts Console messages Causes Encapsulation errors Address mapping errors Framing errors STP failure or loops
Follow the Path
first discovers the actual traffic path all the way from source to destination. The scope of troubleshooting is reduced to just the links and devices that are in the forwarding path. The objective is to eliminate the links and devices that are irrelevant to the troubleshooting task at hand.
End-System Documentation Files
focuses on the hardware and software used in servers, network management consoles, and user workstations. An incorrectly configured end-system can have a negative impact on the overall performance of a network. For this reason, having access to end-system device documentation can be very useful when troubleshooting.
Trace Route Output
generates a list of hops, router IP addresses and the destination IP address that are successfully reached along the path. This list provides important verification and troubleshooting information. If the data reaches the destination, the trace lists the interface on every router in the path. If the data fails at some hop along the way, the address of the last router that responded to the trace is known. This address is an indication of where the problem or security restrictions reside.
Logical IPv4
illustrates how devices are logically connected to the network. This refers to how devices transfer data across the network when communicating with other devices. Symbols are used to represent network components, such as routers, switches, servers, and hosts. Additionally, connections between multiple sites may be shown, but do not represent actual physical locations. Information recorded on this network topology may include the following: Device identifiers IP addresses and prefix lengths Interface identifiers Routing protocols / static routes Layer 2 information (i.e., VLANs, trunks, EtherChannels)
Test Hypothesis
important to assess the impact and urgency of the problem. Could the solution have an adverse effect on other systems or processes? The severity of the problem should be weighed against the impact of the solution. Create a rollback plan identifying how to quickly reverse a solution. This may prove to be necessary if the solution fails. Implement the solution and verify that it has solved the problem.
Physical Layer Troubleshooting
inconvenience users but can impact the productivity of the entire company. Networks that experience these kinds of conditions usually shut down. Because the upper layers of the OSI model depend on the this layer to function Symptoms: Performance lower than baseline Loss of connectivity Network bottlenecks or congestion High CPU utilization rates Console error messages Causes Power related Hardware faults Cabling faults Attenuation Noise Interface configuration errors Exceeding design limits CPU overload
Solve the problem
inform the users and anyone involved in the troubleshooting process that the problem has been resolved.
Analyze Information
interpreted and analyzed gather information using network documentation, network baselines, searching organizational knowledge bases, searching the internet, and talking with other technicians.
Syslog Server
log information regarding configuration changes, ACL violations, interface status, and many other types of events. Cisco devices can send log messages to several different facilities. Event messages can be sent to one or more of the following: Console - Console logging is on by default. Messages log to the console and can be viewed when modifying or testing the router or switch using terminal emulation software while connected to the console port of the network device. Terminal lines - Enabled EXEC sessions can be configured to receive log messages on any terminal lines. Like console logging, this type of logging is not stored by the network device and, therefore, is only valuable to the user on that line. Buffered logging - Buffered logging is a little more useful as a troubleshooting tool because log messages are stored in memory for a time. However, log messages are cleared when the device is rebooted. SNMP traps - Certain thresholds can be preconfigured on routers and other devices. Router events, such as exceeding a threshold, can be processed by the router and forwarded as SNMP traps to an external SNMP network management station. SNMP traps are a viable security logging facility but require the configuration and maintenance of an SNMP system. Syslog - Cisco routers and switches can be configured to forward log messages to an external syslog service. This service can reside on any number of servers or workstations, including Microsoft Windows and Linux-based systems. Syslog is the most popular message logging facility, because it provides long-term log storage capabilities and a central location for all router messages.
Transport Layer Troubleshooting - NAT for IPv4
misconfigured NAT inside, NAT outside, or ACLs. Other issues include interoperability with other network technologies, especially those that contain or derive information from host network addressing in the packet. Common Interoperability Areas BOOTP and DHCP DNS and WINS SNMP Tunneling and encryption protocols
Switch MAC Address Table (show MAC address-table output)
table lists the MAC address connected to each port. Use the show mac address-table command to display the MAC address table on the switch.
Gather Information
targets (i.e., hosts, devices) to be investigated must be identified, access to the target devices must be obtained, and information gathered.
Performance problems
that there is a difference between the expected behavior and the observed behavior, and the system is not functioning as could be reasonably expected.
commands and other tools such as packet captures and device logs.
ping {host | ip-address} Sends an echo request packet to an address, then waits for a reply The host or ip-address variable is the IP alias or IP address of the target system traceroute destination Identifies the path a packet takes through the networks The destination variable is the hostname or IP address of the target system telnet {host | ip-address} Connects to an IP address using the Telnet application Use SSH whenever possible instead of Telnet ssh -l user-id ip-address Connects to an IP address using SSH SSH is more secure than Telnet show ip interface brief show ipv6 interface brief Displays a summary status of all interfaces on a device Useful for quickly identifying IP addressing on all interfaces. show ip route show ipv6 route Displays the current IPv4 and IPv6 routing tables, which contains the routes to all known network destinations show protocols Displays the configured protocols and shows the global and interface-specific status of any configured Layer 3 protocol debug Displays a list of options for enabling or disabling debugging events (must be preformed during normal business hours, creates large amount of console traffic)
Eliminate Possible Causes
progressively eliminating possible causes to eventually identify the most probable cause.
Educated Guess
shoot-from-the-hip troubleshooting approach. This is a less-structured troubleshooting method that uses an educated guess based on the symptoms of the problem.
Step 1 - Verify the Physical Layer
show processes cpu, show memory, show interfaces
Data Measurement commands
show version Displays uptime, version information for device software and hardware. show ip interface [brief] show ipv6 interface [brief] Displays all the configuration options that are set on an interface. Use the brief keyword to only display up/down status of IP interfaces and the IP address of each interface. show interfaces Displays detailed output for each interface. To display detailed output for only a single interface, include the interface type and number in the command (e.g. Gigabit Ethernet 0/0/0). show ip route show ipv6 route Displays the routing table content listing directly connected networks and learned remote networks. Append static, eigrp, or ospf to display those routes only. show cdp neighbors detail Displays detailed information about directly connected Cisco neighbor devices. show arp show ipv6 neighbors Displays the contents of the ARP table (IPv4) and the neighbor table (IPv6). show running-config Displays current configuration. show vlan Displays the status of VLANs on a switch. show port Displays the status of ports on a switch. show tech-support This command is useful for collecting a large amount of information about the device for troubleshooting purposes. It executes multiple show commands which can be provided to technical support representatives when reporting a problem
Comparison
spot-the-differences approach and attempts to resolve the problem by changing the nonoperational elements to be consistent with the working ones. compare configurations, software versions, hardware, or other device properties, links, or processes between working and nonworking situations and spot significant differences between them. The weakness of this method is that it might lead to a working solution, without clearly revealing the root cause of the problem.
Substitution
swap-the-component because you physically swap the problematic device with a known, working one. If the problem is fixed, then the problem is with the removed device. If the problem remains, then the cause may be elsewhere. example, a border router goes down. It may be more beneficial to simply replace the device and restore service, rather than to troubleshoot the issue.
Troubleshoot VLAN
switched network, each port in a switch belongs to a VLAN. Each VLAN is considered a separate logical network, and packets destined for stations that do not belong to the VLAN must be forwarded through a device that supports routing. host in one VLAN sends a broadcast Ethernet frame, such as an ARP request, all hosts in the same VLAN receive the frame; hosts in other VLANs do not. Even if two hosts are in the same IP network, they will not be able to communicate if they are connected to ports assigned to two separate VLANs. Additionally, if the VLAN to which the port belongs is deleted, the port becomes inactive. All hosts attached to ports belonging to the VLAN that was deleted are unable to communicate with the rest of the network. Commands such as show vlan can be used to validate VLAN assignments on a switch. *check arp -a on PC (in example default gateway no longer listed) *check show MAC address-table on switch (shows pc on different vlan) *on switch add interface back to vlan 10 so that pc can find default & communicate. Int fa0/1 switchport mode access swithchport access vlan 10 exit #show mac address-table
Troubleshoot IPv6 Default Gateway
the default gateway can be configured manually, using stateless autoconfiguration (SLAAC), or by using DHCPv6. With SLAAC, the default gateway is advertised by the router to hosts using ICMPv6 Router Advertisement (RA) messages. The default gateway in the RA message is the link-local IPv6 address of a router interface. If the default gateway is configured manually on the host, which is very unlikely, the default gateway can be set to either the global IPv6 address, or to the link-local IPv6 address. show ipv6 route Cisco IOS command is used to check for the IPv6 default route PC1 Addressing ipconfig Windows command is used to verify that a PC1 has an IPv6 default gateway. Check R1 Interface Settings show ipv6 interface GigabitEthernet 0/0/0 on R1 reveals that although the interface has an IPv6 address, it is not a member of the All-IPv6-Routers multicast group Correct R1 IPv6 Routing R1 is enabled as an IPv6 router using the ipv6 unicast-routing command. The show ipv6 interface GigabitEthernet 0/0/0 command verifies that R1 is a member of ff02::2, the All-IPv6-Routers multicast group.
Propose hypothesis
the most probable cause has been identified, a solution must be formulated
physical network diagram
topology shows the physical layout of the devices connected to the network. You need to know how devices are physically connected to troubleshoot physical layer problems. Information recorded on this topology typically includes the following: Device name Device location (address, room number, rack location) Interface and ports used Cable type
bottom-up model
use when the problem is suspected to be a physical one. requires that you check every device and interface on the network until the possible cause of the problem is found. Start at: Physical Layer Data Link Network Transport Application
Application Layer Troubleshooting
used for network management, file transfer, distributed file services, terminal emulation, and email. New user services are often added, such as VPNs and VoIP. SSH/Telnet Enables users to establish terminal session connections with remote hosts. HTTP Supports the exchanging of text, graphic images, sound, video, and other multimedia files on the web. FTP Performs interactive file transfers between hosts. TFTP Performs basic interactive file transfers typically between hosts and networking devices. SMTP Supports basic message delivery services. POP Connects to mail servers and downloads email. SNMP Collects management information from network devices. DNS Maps IP addresses to the names assigned to network devices. Network File System (NFS) Enables computers to mount drives on remote hosts and operate them as if they were local drives. Originally developed by Sun Microsystems, it combines with two other application layer protocols, external data representation (XDR) and remote-procedure call (RPC), to allow transparent access to remote network resources.
Baseline
used to establish normal network or system performance to determine the "personality" of a network under normal conditions. collect performance data from the ports and devices that are essential to network operation.
Ping Output
uses a Layer 3 protocol that is a part of the TCP/IP suite called ICMP. Ping uses the ICMP echo request and ICMP echo reply packets. If the host at the specified address receives the ICMP echo request, it responds with an ICMP echo reply packet. Ping can be used to verify end-to-end connectivity for both IPv4 and IPv6.
Define the Problem
usually identified by a symptom (e.g., the network is slow or has stopped working). Network symptoms may appear in many different forms, including alerts from the network management system, console messages, and user complaints.
Step 3 - Verify Addressing on the Local Network
verify mappings between destination IP addresses and Layer 2 Ethernet addresses on individual segments. In IPv4, this functionality is provided by ARP. In IPv6, the ARP functionality is replaced by the neighbor discovery process and ICMPv6. The neighbor table caches IPv6 addresses and their resolved Ethernet physical (MAC) addresses. Windows IPv4 ARP Table Windows IPv6 Neighbor Table IOS IPv6 Neighbor Table Switch MAC address table
