Chapter 12 - Network Troubleshooting

Look for damaged cables, improper cable, and poorly crimped connectors. Suspect cables should be tested or exchanged with a known functioning cable.

Cabling faults

Caused by a noisy serial line, an improperly designed cable, faulty NIC, duplex mismatch, or an incorrectly configured channel service unit (CSU) line clock.

Framing errors

Determine whether anything in the network has recently changed, and if there is anyone currently working on the network infrastructure.

General network issues

Often a change in the topology may unknowingly have effects on other areas of the network.

General network issues

Seven-Step Troubleshooting Process The goal of this stage is to verify that there is a problem and then properly define what the problem is.

1. Define the problem

Sends an echo request packet to an address, then waits for a reply.

ping {host | ip-address}

Handheld devices are designed for testing the various types of data communication cabling.

Cable Testers

Performs basic interactive file transfers, typically between hosts and networking devices.

TFTP

Connects to an IP address using the Telnet Application

telnet {host | ip-address}

NetBIOS name service port

137

NetBIOS datagram service Port

138

Seven-Step Troubleshooting Process In this step, targets (i.e., hosts, devices) to be investigated must be identified, access to the target devices must be obtained, and information gathered. During this step, the technician may gather and document more symptoms, depending on the characteristics that are identified.

2. Gather Information

Seven-Step Troubleshooting Process Possible causes must be identified. The gathered information is interpreted and analyzed using network documentation, network baselines, searching organizational knowledge bases, searching the internet, and talking with other technicians.

3. Analyze Information

Time port

37

Seven-Step Troubleshooting Process If multiple causes are identified, then the list must be reduced by progressively eliminating possible causes to eventually identify the most probable cause. Troubleshooting experience is extremely valuable to quickly eliminate causes and identify the most probable cause.

4. Eliminate Possible Causes

TACACS port

49

Seven-Step Troubleshooting Process When the most probable cause has been identified, a solution must be formulated. At this stage, troubleshooting experience is very valuable when proposing a plan.

5. Propose Hypothesis

DNS port

53

Seven-Step Troubleshooting Process Before testing the solution, it is important to assess the impact and urgency of the problem. For instance, could the solution have an adverse effect on other systems or processes? The severity of the problem should be weighed against the impact of the solution. For example, if a critical server or router must be offline for a significant amount of time, it may be better to wait until the end of the workday to implement the fix. Sometimes, a workaround can be created until the actual problem is resolved. Create a rollback plan identifying how to quickly reverse a solution. This may prove to be necessary if the solution fails.

6. Test Hypothesis

BOOTP/DHCP server port

67

BOOTP/DHCP client port

68

TFTP port

69

Seven-Step Troubleshooting Process When the problem is solved, inform the users and anyone involved in the troubleshooting process that the problem has been resolved. Other IT team members should be informed of the solution. Appropriate documentation of the cause and the fix will assist other support technicians in preventing and solving similar problems in the future.

7. Solve the problem

Complex IPv4 wildcard masks are more efficient, but are more subject to configuration errors.

Addresses and IPv4 wildcard masks

This can be caused if a cable length exceeds the design limit for the media, or when there is a poor connection resulting from a loose cable, or dirty/oxidized contacts.

Attenuation

NAT Issues - Configuring the IPv4 helper feature can help solve this problem.

BOOTP and DHCP DNS SNMP

NAT Issues - The DHCP-Request packet has a source IPv4 address of 0.0.0.0.

Boot and DHCP

Good approach to use when the problem is suspected to be a physical one.

Bottom-up

Logging is a little more useful as a troubleshooting tool because log messages are stored in memory for a time. However, log messages are cleared when the device is rebooted.

Buffered

Symptoms include processes with high CPU utilization percentages, input queue drops, slow performance, SNMP timeouts, no remote access, no DHCP services, Telnet, and pings ae slow or fail to respond.

CPU overload

Multifunctional handheld devices used to test and certify copy and fiber cables.

Cable Analyzers

Browser-based interface that displays device performance analysis in a switched and routed environment.

Cisco Prime NAM

Attempts to resolve the problem by comparing a non-operational element with the working one.

Comparison

Check for any equipment and connectivity problems, including power problems, environmental problems, and Layer 1 problem, such as cabling problems, bad ports, and ISP problems.

Connectivity issues

This is on by default. Messages log to the console and can be viewed when modifying or testing the router or switch using terminal emulation software while connected to the console port of the network device.

Console logging

Routers send messages when it detects a problem when keepalives are expected but do not arrive.

Console messages

The most common console message that indicates a Layer 2 problem is a line protocol down message.

Console messages.

Maps IP addresses to the names assigned to network devices

DNS

NAT Issues - A server outside the NAT router does not have an accurate representation of the network inside the router.

DNS

Devices measure electrical values of voltage, current, and resistance.

Digital Multimeters

Start at the middle layer (Layer 3) and test in both directions from that layer.

Divide and Conquer

Success of this method varies based on your troubleshooting experience and ability.

Educated guess

Occurs when bits placed in a field by the sender are not what the receiver expects to see (integrity checking, FCS).

Encapsulation errors

A component could operate sub-optimally if it is being utilized beyond specifications.

Exceeding design limits

Operating systems use broadcasts and multicasts extensively.

Excessive broadcasts

These are the result of an overly large Layer 2 broadcast domain.

Excessive broadcasts

Performs interactive file transfers between hosts.

FTP

Used to discover the actual traffic path from source to destination to reduce the scope of troubleshooting.

Follow the Path

Supports the exchanging of text, graphic images, sound, video, and other multimedia files on the web.

HTTP

Supports the exchanging of text, graphic images, sound, video, and other multimedia files on the web. - Securely

HTTPS

Faulty or corrupt NIC driver files, bad cabling, or grounding problems can cause network transmission errors such as late collisions, short frames, and jabber.

Hardware faults

The implicit ACE can be the cause of an ACL misconfiguration.

Implicit deny any

Causes can include incorrect clock rate, incorrect clock source, and interface not being turned on. This causes a loss of connectivity with attached network segments.

Interface configuration errors

When vendor-based knowledge bases are combined with internet search engines, a network administrator has access to a vast pool of experience-based information.

Knowledge Bases

Enables computers to mount and use drives on remote hosts.

NFS

Some Layer 2 problems can stop the exchange of frames across a link, while others only cause network performance to degrade (FCS failures and TCP resends).

No functionality or connectivity at the network layer or above.

Check to see if there are any problems with the routers forming neighbor adjacencies.

Neighbor issues

Network software include device-level monitoring, configuration, and fault management tools.

Network Management System Tools

An extended or continuous ping can help reveal if frames are being dropped.

Network is operating below baseline levels

Frames can take a suboptimal path to their destination but still arrive causing the network to experience unexpected high-bandwidth usage on links.

Network is operating below baseline performance levels.

Local electromagnetic interference (EMI) can be generated by many sources, such as crosstalk, nearby electric cables, large electric motors, FM radio stations, police radio, and more.

Noise

The entries in an ACL should be from specific to general.

Order of access control entries

Connects to mail servers and downloads email.

POP

Specialized device used for troubleshooting switched networks and VLANs. Detect, diagnose, and resolve network performance issues. Track response time, availability, and uptime of routers, switches, and other SNMP-enabled devices. Monitor and analyze network bandwidth performance and traffic patterns. Find bandwidth hogs on a network and see which applications are using the most bandwidth

Portable Network Analyzers

Check the operation of the fans and ensure that the chassis intake and exhaust vents are clear.

Power-related

Check the routing table for anything unexpected, such as missing routes or unexpected routes.

Routing table

Supports basic message delivery services

SMTP

Collects management information from network devices.

SNMP

NAT Issues - A management station on one side of a NAT router may not be able to contact agents on the other side of the NAT router.

SNMP

Certain thresholds can be preconfigured on routers and other devices. Router events, such as exceeding a threshold, can be processed by the router and forwarded to an external network management station. Messages are a viable security logging facility but require the configuration and maintenance.

SNMP traps

Enables users to establish terminal session connections with remote hosts.

SSH/Telnet

Most of these problems are related to forwarding loops that occur when no ports in a redundant topology are blocked and traffic is forwarded in circles indefinitely, excessive flooding because of a high rate of topology changes.

STP failures/loops

An ACL must be applied to the correct interface in the correct direction.

Selection of traffic flow

It is important that only the correct transport layer protocol be specified in an ACE.

Selection of transport layer protocol

Ensuring that the correct inbound and outbound ports are specified in an ACE

Source and destination ports

You physically swap a suspected problematic device with a known, working one.

Substitution

Cisco routers and switches can be configured to forward log messages to an external syslog service. This service can reside on any number of servers or workstations, including Microsoft Windows and Linux-based systems. The most popular message logging facility, because it provides long-term log storage capabilities and a central location for all router messages.

Syslog

Enabled EXEC sessions can be configured to receive log messages on any terminal lines. Like console logging, this type of logging is not stored by the network device and, therefore, is only valuable to the user on that line.

Terminal lines

Use this approach for simpler problems, or when you think the problem is with a piece of software.

Top-down

Check the table for anything unexpected, such as missing entries or unexpected entries.

Topology database

NAT Issues - Protocols often require that traffic be sourced from a specific UDP/TCP port, or use a protocol a the transport layer that cannot be processed by NAT.

Tunneling and Encryption protocols

Misconfigured ACLs often cause problems for protocols other than TCP and UDP.

Uncommon protocols

The established keyword applied incorrectly, can provide unexpected results.

Use of the established keyword

Cisco NAM-3 Utilization

Voice and video quality Traffic Application performance Packet capturing for troubleshooting URL monitoring for web filtering policies & QoS Application and host analysis in VLAN

Displays a list of options for enabling or disabling debugging events

debug

Displays detailed information about directly connected Cisco neighbor devices.

show cdp neighbors detail

Displays a summary status of all interfaces on a device.

show ip interface brief show ipv6 interface brief

Displays the current IPv4 and IPv6 routing tables.

show ip route show ipv6 route

Displays the global and interface-specific status of any configured Layer 3 protocol.

show protocols

This command is useful for collecting a large amount of information about the device for troubleshooting purposes. It executes multiple show commands which can be provided to technical support representatives when reporting a problem

show tech-support

Connects to an IP address using SSH.

ssh -l user-id ip-address

Identifies the path a packet takes through the networks.

traceroute destination