Chapter 8: Network Troubleshooting

physical, type

A ______________ network topology shows the physical layout of the devices connected to the network. It is necessary to know how devices are physically connected to troubleshoot physical layer problems. Information recorded on the diagram typically includes: Device ___________ Model and manufacturer Operating system version Cable type and identifier Cable specification Connector type Cabling endpoints

logical, identifiers

A ___________________ network topology illustrates how devices are logically connected to the network, meaning how devices actually transfer data across the network when communicating with other devices. Symbols are used to represent network elements, such as routers, servers, hosts, VPN concentrators, and security devices. Additionally, connections between multiple sites may be shown, but do not represent actual physical locations. Information recorded on a logical network diagram may include: Device ____________ IP address and prefix lengths Interface identifiers Connection type Frame Relay DLCI for virtual circuits (if applicable) Site-to-site VPNs Routing protocols Static routes Data-link protocols WAN technologies used

baseline

A ____________________ is used to establish normal network or system performance. Establishing a network performance baseline requires collecting performance data from the ports and devices that are essential to network operation. The figure shows several questions that a baseline should answer.

logical

After all symptoms are gathered, if no solution is identified, the network administrator compares the characteristics of the problem to the ____________ layers of the network to isolate and solve the issue.

versions

Another approach involves comparing a working and non-working situation, and spotting significant differences, including: Configurations Software ____________ Hardware and other device properties Using this method may lead to a working solution, but without clearly revealing the cause of the problem. This method can be helpful when the network administrator is lacking an area of expertise, or when the problem needs to be resolved quickly. After the fix has been implemented, the network administrator can do further research on the actual cause of the problem.

version, neighbors

Commands used for Data Collection: *Brief makes it show only the up/down status of the IP interfaces and the IP address of each interface* show ___________ show ip interface [brief] show ipv6 interface [brief] show interfaces [interface_type interface_number] Example: show interface gigabitEthernet 0/0 show ip route show ipv6 route show arp (ARP table for IPv4) show ipv6 _____________ (Nieghbor table for IPv6) show running-config show port (Status port on a switch) show vlan (Status of VLANs on a Switch) show tech-support (Executes multiple show commands) show ip cache flow (Displays a summary of the NetFlow accounting statistics)

1. Gather Information 2. Determine Ownership 3. Narrow the scope 4. Gather symptoms from suspected devices. 5. Document Symptoms

Five Steps to Gather Information:

Configuration files, including network configuration files and end-system configuration files Physical and logical topology diagrams A baseline performance levels

For network administrators to be able to monitor and troubleshoot a network, they must have a complete set of accurate and current network documentation. This documentation includes (3):

DNS

For troubleshooting purposes, the following information could be documented within the end-system configuration table: - Device name (purpose) - Operating system and version - IPv4 and IPv6 addresses - Subnet mask and prefix length - Default gateway and _________ server - Any high-bandwidth network applications used on the end system

traffic

IP SLAs use generated ____________ to measure network performance between two networking devices, multiple network locations, or across multiple network paths. In the example in the figure, R1 is the IP SLA source that monitors the connection to the DNS server by periodically sending ICMP requests to the server.

show arp show vlan show ip cache flow show running-configuration

Identify Commands used for Measuring Data: _______________ shows contents of the address resolution table. _______________ shows summary of VLANs and access ports on a switch. ______________ shows summary of the Netflow accounting statistics. ______________ shows the current configuration of the device.

show version command show interfaces command show ip route command show ip interface brief

Identify Commands used for Measuring Data: _________________ is used to show uptime about a device software and hardware. ______________ is used to show detailed settings and status for device interfaces. _______________ shows contents of the routing table. ________________ shows summarized table of the up/down status of all device interfaces.

Bottom-up

In __________________ troubleshooting, you start with the physical components of the network and move up through the layers of the OSI model until the cause of the problem is identified. The disadvantage with the ____________ troubleshooting approach is it requires that you check every device and interface on the network until the possible cause of the problem is found.

divide-and-conquer

In ______________________ troubleshooting, you start by collecting user experiences of the problem, document the symptoms and then, using that information, make an informed guess as to which OSI layer to start your investigation. When a layer is verified to be functioning properly, it can be assumed that the layers below it are functioning. The administrator can work up the OSI layers. If an OSI layer is not functioning properly, the administrator can work down the OSI layer model. For example, if users cannot access the web server, but they can ping the server, then the problem is above Layer 3. If pinging the server is unsuccessful, then the problem is likely at a lower OSI layer.

educated guess

In addition to the systematic, layered approach to troubleshooting, there are also, less-structured troubleshooting approaches. One troubleshooting approach is based on an _________________ by the network administrator, based on the symptoms of the problem. This method is more successfully implemented by seasoned network administrators, because seasoned network administrators rely on their extensive knowledge and experience to decisively isolate and solve network issues. With a less-experienced network administrator, this troubleshooting method may be more like random troubleshooting.

end

In many cases the problem is reported by an end user. The information may often be vague or misleading, such as, "The network is down" or "I cannot access my email". In these cases, the problem must be better defined. This may require asking questions of the ______ users.

Shoot from the Hip

Key words to know - Bottom-up - Top-Down - Divide & Conquer - Shoot from the Hip - Spot the difference - Substitution Which troubleshooting method is used (listed above) - Use an experienced troubleshooting guess to investigate a possible cause.

OSI and TCP/IP Models

Logical networking models, such as the ________ and _________ models, separate network functionality into modular layers. These layered models can be applied to the physical network to isolate network problems when troubleshooting. For example, if the symptoms suggest a physical connection problem, the network technician can focus on troubleshooting the circuit that operates at the physical layer. If that circuit functions as expected, the technician looks at areas within another layer that could be causing the problem.

location

Network Configuration Files Information that could be captured within a device table includes: - Type of device, model designation - IOS image name - Device network hostname - ____________ of the device (building, floor, room, rack, panel) - If modular, include each module type and slot number - Data link layer addresses - Network layer addresses - Any additional important information about physical aspects of the device

Service Level Agreement (SLA)

Network administrators must be proactive and continually monitor and test the network. The goal is to discover a network failure as early as possible. A useful tool for this task is the Cisco IOS IP ___________________________.

hard copy

Network documentation allows network administrators to efficiently diagnose and correct network problems, based on the network design and the expected performance of the network under normal operating conditions. All network documentation information should be kept in a single location either as _______________ or on the network on a protected server. Backup documentation should be maintained and kept in a separate location.

real-time

Network engineers use IP SLAs to simulate network data and IP services to collect network performance information in _________________. Performance monitoring can be done anytime, anywhere, without deploying a physical probe.

physical, logical

Network topology diagrams keep track of the location, function, and status of devices on the network. There are two types of network topology diagrams: the __________ topology and the ___________ topology.

debug

Note: Although the ______________ command is an important tool for gathering symptoms, it generates a large amount of console message traffic and the performance of a network device can be noticeably affected. If the debug must be performed during normal working hours, warn network users that a troubleshooting effort is underway and that network performance may be affected. Remember to disable debugging when you are done

ping, traceroute

Note: ______________ and ______________ are probe tools. A physical probe is different. It is a device that can be inserted somewhere in the network to collect and monitor traffic. The use of physical probes is beyond the scope of this course.

3

Notice that routers and multilayer switches are shown at Layer 4, the transport layer. Although routers and multilayer switches usually make forwarding decisions at Layer ____, ACLs on these devices can be used to make filtering decisions using Layer 4 information.

network management

Sophisticated ___________________________ software is typically used to baseline large and complex networks. These software packages enable administrators to automatically create and review reports, compare current performance levels with historical observations, automatically identify performance problems, and create alerts for applications that do not provide expected levels of service.

addressing

The TCP/IP Internet layer relates to the OSI network layer. The Internet layer is responsible for _______________ used for data transfer from source to destination.

physical, data link

The TCP/IP network access layer corresponds to the OSI ______________ and ____________ layers. The network access layer communicates directly with the network media and provides an interface between the architecture of the network and the Internet layer.

session, presentation, application

The application layer in the TCP/IP suite actually combines the functions of the three OSI model layers: ___________, ______________, and _______________. The application layer provides communication between applications, such as FTP, HTTP, and SMTP on separate hosts.

1 to 4

The lower layers (_____to_____) of the OSI model handle data-transport issues. Layers 3 and 4 are generally implemented only in software. The physical layer (Layer 1) and data link layer (Layer 2) are implemented in hardware and software. The physical layer is closest to the physical network medium, such as the network cabling, and is responsible for actually placing information on the medium.

segments

The transport layers of TCP/IP and OSI directly correspond in function. The transport layer is responsible for exchanging ________________ between devices on a TCP/IP network.

5 to 7

The upper layers (____ to _____) of the OSI model deal with application issues and generally are implemented only in software. The application layer is closest to the end user. Both users and application layer processes interact with software applications that contain a communications component.

Stage 1. Gather symptoms Stage 2. Isolate the problem Stage 3. Implement corrective action Problem fixed? Yes or no? If fixed, document solution and save changes. If it did not fix the problem or if it created another problem, undo corrective action and start again.

There are three major stages to the troubleshooting process:

Step 1. Determine what types of data to collect - When conducting the initial baseline, start by selecting a few variables that represent the defined policies Step 2. Identify devices and ports of interest. Use the network topology to identify those devices and ports for which performance data should be measured (Servers, Key Users, Network device ports that connect to other network devices.) Step 3. Determine the baseline duration. The length of time and the baseline information being gathered must be sufficient for establishing a typical picture of the network. When capturing data for analysis, the period specified should be, at a minimum, seven days long.

To establish and capture an initial network baseline, perform the following steps (3):

six

Typically, a baseline needs to last no more than _____ weeks, unless specific long-term trends need to be measured. Generally, a two-to-four-week baseline is adequate.

Bottom-up Top-down Divide-and-conquer

Using the layered models, there are three primary methods for troubleshooting networks:

Gathering Symptoms

When _____________________, it is important that the administrator gather facts and evidence to progressively eliminate possible causes, and eventually identify the root cause of the issue. By analyzing the information, the network administrator formulates a hypothesis to propose possible causes and solutions, while eliminating others.

ip interface brief, ip route, cdp neighbors

When documenting the network, it is often necessary to gather information directly from routers and switches. Obvious useful network documentation commands include ping, traceroute, and telnet as well as the following show commands: - The show ___________ and show ipv6 interface brief commands are used to display the up or down status and IP address of all interfaces on a device. - The show__________ and show ipv6 route commands are used to display the routing table in a router to learn the directly connected neighbors, more remote devices (through learned routes), and the routing protocols that have been configured. -The show _____________ detail command is used to obtain detailed information about directly connected Cisco neighbor devices.

Layer 4 - Transport

Which layer is this issue at? ACLs are misconfigured and blocking all web traffic

Layer 4 - Transport

Which layer is this issue at? SNMP messages are unable to traverse NAT.

Layer 5,6,7

Which layer is this issue at? SSH error messages display unknown/untrusted certificates.

Layer 2 - Data Link

Which layer is this issue at? STP loops and route flapping are generating a broadcast storm.

Layer 5,6,7

Which layer is this issue at? The DNS server is not configured with correct URLs .

Layer 3

Which layer is this issue at? The routing table is missing routes and has unknown network listed.

Layer 1 - Physical

Which layer is this issue at? Traffic is congested on a low capacity link and frames are lost.

Network

____________ configuration files contain accurate, up-to-date records of the hardware and software used in a network. Within the network configuration files a table should exist for each network device used on the network, containing all relevant information about that device.

Analysis

________________ after an initial baseline also tends to reveal hidden problems. The collected data shows the true nature of congestion or potential congestion in a network. It may also reveal areas in the network that are underutilized and quite often can lead to network redesign efforts, based on quality and capacity observations.

Top-down

________________ troubleshooting starts with the end-user applications and moves down through the layers of the OSI model until the cause of the problem has been identified. End-user applications of an end system are tested before tackling the more specific networking pieces. Use this approach for simpler problems, or when you think the problem is with a piece of software. The disadvantage with the top-down approach is it requires checking every network application until the possible cause of the problem is found

End-system

____________________ configuration files focus on the hardware and software used in end-system devices, such as servers, network management consoles, and user workstations. An incorrectly configured end system can have a negative impact on the overall performance of a network. For this reason, having a sample baseline record of the hardware and software used on devices, and recorded in end-system documentation as shown in Figure 3 can be very useful when troubleshooting.

Substitution

____________________ is another quick troubleshooting methodology. It involves swapping the problematic device with a known, working one. If the problem is fixed, that the network administrator knows the problem is with the removed device. If the problem remains, then the cause may be elsewhere. In specific situations, this can be an ideal method for quick problem resolution, such as when a critical single point of failure, like a border router, goes down. It may be more beneficial to simply replace the device and restore service, rather than troubleshoot the issue.