Chapter 12 - Network Troubleshooting
Look for damaged cables, improper cable, and poorly crimped connectors. Suspect cables should be tested or exchanged with a known functioning cable.
Cabling faults
Caused by a noisy serial line, an improperly designed cable, faulty NIC, duplex mismatch, or an incorrectly configured channel service unit (CSU) line clock.
Framing errors
Determine whether anything in the network has recently changed, and if there is anyone currently working on the network infrastructure.
General network issues
Often a change in the topology may unknowingly have effects on other areas of the network.
General network issues
Seven-Step Troubleshooting Process The goal of this stage is to verify that there is a problem and then properly define what the problem is.
1. Define the problem
Sends an echo request packet to an address, then waits for a reply.
ping {host | ip-address}
Handheld devices are designed for testing the various types of data communication cabling.
Cable Testers
Performs basic interactive file transfers, typically between hosts and networking devices.
TFTP
Connects to an IP address using the Telnet Application
telnet {host | ip-address}
NetBIOS name service port
137
NetBIOS datagram service Port
138
Seven-Step Troubleshooting Process In this step, targets (i.e., hosts, devices) to be investigated must be identified, access to the target devices must be obtained, and information gathered. During this step, the technician may gather and document more symptoms, depending on the characteristics that are identified.
2. Gather Information
Seven-Step Troubleshooting Process Possible causes must be identified. The gathered information is interpreted and analyzed using network documentation, network baselines, searching organizational knowledge bases, searching the internet, and talking with other technicians.
3. Analyze Information
Time port
37
Seven-Step Troubleshooting Process If multiple causes are identified, then the list must be reduced by progressively eliminating possible causes to eventually identify the most probable cause. Troubleshooting experience is extremely valuable to quickly eliminate causes and identify the most probable cause.
4. Eliminate Possible Causes
TACACS port
49
Seven-Step Troubleshooting Process When the most probable cause has been identified, a solution must be formulated. At this stage, troubleshooting experience is very valuable when proposing a plan.
5. Propose Hypothesis
DNS port
53
Seven-Step Troubleshooting Process Before testing the solution, it is important to assess the impact and urgency of the problem. For instance, could the solution have an adverse effect on other systems or processes? The severity of the problem should be weighed against the impact of the solution. For example, if a critical server or router must be offline for a significant amount of time, it may be better to wait until the end of the workday to implement the fix. Sometimes, a workaround can be created until the actual problem is resolved. Create a rollback plan identifying how to quickly reverse a solution. This may prove to be necessary if the solution fails.
6. Test Hypothesis
BOOTP/DHCP server port
67
BOOTP/DHCP client port
68
TFTP port
69
Seven-Step Troubleshooting Process When the problem is solved, inform the users and anyone involved in the troubleshooting process that the problem has been resolved. Other IT team members should be informed of the solution. Appropriate documentation of the cause and the fix will assist other support technicians in preventing and solving similar problems in the future.
7. Solve the problem
Complex IPv4 wildcard masks are more efficient, but are more subject to configuration errors.
Addresses and IPv4 wildcard masks
This can be caused if a cable length exceeds the design limit for the media, or when there is a poor connection resulting from a loose cable, or dirty/oxidized contacts.
Attenuation
NAT Issues - Configuring the IPv4 helper feature can help solve this problem.
BOOTP and DHCP DNS SNMP
NAT Issues - The DHCP-Request packet has a source IPv4 address of 0.0.0.0.
Boot and DHCP
Good approach to use when the problem is suspected to be a physical one.
Bottom-up
Logging is a little more useful as a troubleshooting tool because log messages are stored in memory for a time. However, log messages are cleared when the device is rebooted.
Buffered
Symptoms include processes with high CPU utilization percentages, input queue drops, slow performance, SNMP timeouts, no remote access, no DHCP services, Telnet, and pings ae slow or fail to respond.
CPU overload
Multifunctional handheld devices used to test and certify copy and fiber cables.
Cable Analyzers
Browser-based interface that displays device performance analysis in a switched and routed environment.
Cisco Prime NAM
Attempts to resolve the problem by comparing a non-operational element with the working one.
Comparison
Check for any equipment and connectivity problems, including power problems, environmental problems, and Layer 1 problem, such as cabling problems, bad ports, and ISP problems.
Connectivity issues
This is on by default. Messages log to the console and can be viewed when modifying or testing the router or switch using terminal emulation software while connected to the console port of the network device.
Console logging
Routers send messages when it detects a problem when keepalives are expected but do not arrive.
Console messages
The most common console message that indicates a Layer 2 problem is a line protocol down message.
Console messages.
Maps IP addresses to the names assigned to network devices
DNS
NAT Issues - A server outside the NAT router does not have an accurate representation of the network inside the router.
DNS
Devices measure electrical values of voltage, current, and resistance.
Digital Multimeters
Start at the middle layer (Layer 3) and test in both directions from that layer.
Divide and Conquer
Success of this method varies based on your troubleshooting experience and ability.
Educated guess
Occurs when bits placed in a field by the sender are not what the receiver expects to see (integrity checking, FCS).
Encapsulation errors
A component could operate sub-optimally if it is being utilized beyond specifications.
Exceeding design limits
Operating systems use broadcasts and multicasts extensively.
Excessive broadcasts
These are the result of an overly large Layer 2 broadcast domain.
Excessive broadcasts
Performs interactive file transfers between hosts.
FTP
Used to discover the actual traffic path from source to destination to reduce the scope of troubleshooting.
Follow the Path
Supports the exchanging of text, graphic images, sound, video, and other multimedia files on the web.
HTTP
Supports the exchanging of text, graphic images, sound, video, and other multimedia files on the web. - Securely
HTTPS
Faulty or corrupt NIC driver files, bad cabling, or grounding problems can cause network transmission errors such as late collisions, short frames, and jabber.
Hardware faults
The implicit ACE can be the cause of an ACL misconfiguration.
Implicit deny any
Causes can include incorrect clock rate, incorrect clock source, and interface not being turned on. This causes a loss of connectivity with attached network segments.
Interface configuration errors
When vendor-based knowledge bases are combined with internet search engines, a network administrator has access to a vast pool of experience-based information.
Knowledge Bases
Enables computers to mount and use drives on remote hosts.
NFS
Some Layer 2 problems can stop the exchange of frames across a link, while others only cause network performance to degrade (FCS failures and TCP resends).
No functionality or connectivity at the network layer or above.
Check to see if there are any problems with the routers forming neighbor adjacencies.
Neighbor issues
Network software include device-level monitoring, configuration, and fault management tools.
Network Management System Tools
An extended or continuous ping can help reveal if frames are being dropped.
Network is operating below baseline levels
Frames can take a suboptimal path to their destination but still arrive causing the network to experience unexpected high-bandwidth usage on links.
Network is operating below baseline performance levels.
Local electromagnetic interference (EMI) can be generated by many sources, such as crosstalk, nearby electric cables, large electric motors, FM radio stations, police radio, and more.
Noise
The entries in an ACL should be from specific to general.
Order of access control entries
Connects to mail servers and downloads email.
POP
Specialized device used for troubleshooting switched networks and VLANs. Detect, diagnose, and resolve network performance issues. Track response time, availability, and uptime of routers, switches, and other SNMP-enabled devices. Monitor and analyze network bandwidth performance and traffic patterns. Find bandwidth hogs on a network and see which applications are using the most bandwidth
Portable Network Analyzers
Check the operation of the fans and ensure that the chassis intake and exhaust vents are clear.
Power-related
Check the routing table for anything unexpected, such as missing routes or unexpected routes.
Routing table
Supports basic message delivery services
SMTP
Collects management information from network devices.
SNMP
NAT Issues - A management station on one side of a NAT router may not be able to contact agents on the other side of the NAT router.
SNMP
Certain thresholds can be preconfigured on routers and other devices. Router events, such as exceeding a threshold, can be processed by the router and forwarded to an external network management station. Messages are a viable security logging facility but require the configuration and maintenance.
SNMP traps
Enables users to establish terminal session connections with remote hosts.
SSH/Telnet
Most of these problems are related to forwarding loops that occur when no ports in a redundant topology are blocked and traffic is forwarded in circles indefinitely, excessive flooding because of a high rate of topology changes.
STP failures/loops
An ACL must be applied to the correct interface in the correct direction.
Selection of traffic flow
It is important that only the correct transport layer protocol be specified in an ACE.
Selection of transport layer protocol
Ensuring that the correct inbound and outbound ports are specified in an ACE
Source and destination ports
You physically swap a suspected problematic device with a known, working one.
Substitution
Cisco routers and switches can be configured to forward log messages to an external syslog service. This service can reside on any number of servers or workstations, including Microsoft Windows and Linux-based systems. The most popular message logging facility, because it provides long-term log storage capabilities and a central location for all router messages.
Syslog
Enabled EXEC sessions can be configured to receive log messages on any terminal lines. Like console logging, this type of logging is not stored by the network device and, therefore, is only valuable to the user on that line.
Terminal lines
Use this approach for simpler problems, or when you think the problem is with a piece of software.
Top-down
Check the table for anything unexpected, such as missing entries or unexpected entries.
Topology database
NAT Issues - Protocols often require that traffic be sourced from a specific UDP/TCP port, or use a protocol a the transport layer that cannot be processed by NAT.
Tunneling and Encryption protocols
Misconfigured ACLs often cause problems for protocols other than TCP and UDP.
Uncommon protocols
The established keyword applied incorrectly, can provide unexpected results.
Use of the established keyword
Cisco NAM-3 Utilization
Voice and video quality Traffic Application performance Packet capturing for troubleshooting URL monitoring for web filtering policies & QoS Application and host analysis in VLAN
Displays a list of options for enabling or disabling debugging events
debug
Displays detailed information about directly connected Cisco neighbor devices.
show cdp neighbors detail
Displays a summary status of all interfaces on a device.
show ip interface brief show ipv6 interface brief
Displays the current IPv4 and IPv6 routing tables.
show ip route show ipv6 route
Displays the global and interface-specific status of any configured Layer 3 protocol.
show protocols
This command is useful for collecting a large amount of information about the device for troubleshooting purposes. It executes multiple show commands which can be provided to technical support representatives when reporting a problem
show tech-support
Connects to an IP address using SSH.
ssh -l user-id ip-address
Identifies the path a packet takes through the networks.
traceroute destination