5.1: Network Troubleshooting Methodology, 5.2: Hardware & Software Tools, 5.2: Command Line Tools, 5.3: Common Wired Connectivity and Performance Issues, 5.4: Common Wireless Connectivity and Performance Issues, 5.5: Common Network Service Issues
Attenuation
Loss of power in a signal as it travels from the sending device to the receiving device. Decibels are used to measure signal strength.
pathping
A TCP / IP command that provides information about latency and packet loss on a network. Included in Windows NT and later. First phase: Builds a map by running traceroute Second phase: Measures round trip time and packet loss at each hop.
ifconfig
A TCP/IP configuration and management utility used with UNIX and Linux systems.
tcpdump
A command-line protocol analyzer. Administrators use it to capture packets. Available in most unix/linux distros. Included with Mac, available for Windows (WinDump). Dump is written in a standard pcap format (readable by things like Wireshark)
Latency
A delay between the request and the response. Common to use packet captures and protocol analyzers to troubleshoot.
Packet Sniffer
A device or program that monitors network communications and captures data.
Light meter
A device to measure light intensity. Good for validating optical cables.
nslookup
Command-line program in Windows used to determine exactly what information the DNS server is providing about a specific host name. Deprecated (use dig instead)
Duplicate MAC Addresses
Don't see this problem very often May be a misconfiguration, manufacturering error, a man in the middle attack Ping the IP of the device and then check the ARP cache
Multimeter
A measuring instrument for current, AC & DC voltage, and resistance
nmap
A network utility designed to scan a network and create a map. Frequently used as a vulnerability/port scanner. Identifies open ports. Discover OS without logging into a device. Service scan. Includes its own scripting languag (NSE=Nmap Scripting Engine)
Tone generator
A small electronic device that issues a signal on a wire pair. When used in conjunction with a tone locator, it can help locate the termination of a wire pair.
Spectrum Analyzer
A software tool that assesses the characteristics (for example, frequency, amplitude, and the effects of interference) of wireless signals.
Cable tester
A tool for testing if a cable can connect properly end-to-end and to determine if a cable has a short (continuity test). These tools are available for a variety of cable types.
Crimper
A tool used to attach a terminator or connector to the end of a cable. Often includes a wire stripper.
Punchdown Tool
A tool used to punch individual wires from a network cable into their slots to terminate the cable. 66 and 110 blocks most commonly.
Port Scanner
A type of software that searches a network host for open ports (vulnerabilities). Can be used to visually map the network.
arp
Address Resolution Protocol. Determine a MAC address based on an IP address arp -a: View local arp table/cache
Distance Limitations
All signals degrade -Electrical or optical All devices must adhere to a standard Almost always test in advance to measure signal
Blocked TCP/UDP Ports
Applications not working FIrewall or ACL configuration Confirm with a packet capture Run a TCP or UDP-based traceroute tool
Incorrect host-based firewall settings
Applications not working (based on the application in use) Check the host-based firewall settings Take a packet capture from an external device
Untrusted SSL Certificate
Browsers trust certificates from certain CAs Certificate was signed by a CA not on the list If it's an internal server, add your internal CA certificate to the list
Incorrect Netmask
Check your IP configuration, check your documentation Check devices around you
Exhausted DHCP Scope
Clients receive APIPA address (local subnet communication only) Check the DHCP server (add more IP addresses to the pool if possible) IP address management (IPAM) may help; monitor and report on shortages Lower the lease time if you have a lot of transient users
Damaged cables
Copper cables are pretty rugged but not indestructible Can be stepped on or bent Check your physical layer Use a TDR to check within the cable
Expired IP Address
DHCP address should renew before lease expires Check 169.254.*.* for an APIPA assigned address Check the status of your DHCP server
Latency
Delay between transmission and response
tracert (Windows) traceroute (Linux/Mac)
Determine the route a packet takes to a destination. Takes advantage of ICMP Time to Live Exceeded error message (TTL=1 is first router, TTL=2 is second router, etc.) Not all devices will reply with a TTL exceeded message (e.g. firewalls will filter them)
Jitter
Deviation from a predictable data stream
Frequency Mismatch
Devices have to match the access point's available frequencies Verify the client is communicating on the appropriate channel Older standards may slowdown a newer network (802.11b compatibility mode on 802.11g devices)
dig
Domain Information Groper Lookup information from DNS servers. More advanced domain info than nslookup. Not native to any Windows distributions but you can download it.
Channel Overlap
Easier on 5GHz since you have more channels to work with
EMI
Electromagnetic Interference Watch your bend radius, watch your cable ties, don't twist them Avoid power cables, fluorescent lights, fire prevention components Test after installation with a TDR
Security Type Mismatch
Encryption on wireless is important Needs to match between client and access point Migrate all WEP and WPA to WPA2!
Wrong SSID
Every access point has at least one Service Set Identifier (SSID) Confirm the correct SSID settings
Protocol Analyzer
Hardware or software that captures packets to decode and analyze their contents. Good for viewing traffic patterns and doing big data analytics.
Rogue DHCP Server
IP addresses assigned by a non-authorized server Client is assigned an invalid or duplicate address Disable rougue DHCP communication Enable DHCP snooping on your switch Make sure authorized DHCP servers are on Active Directory Release IP addresses and renew with the correct DHCP servers
Power Levels
Increase the power level on an access point to increase its range.
Wrong Passphrase
Is not connected, check the authentication Shared passphrase common in SOHO, not enteprise 802.1X is commonly used in enterprise settings; make sure client is configured to run 802.1X
Incorrect cable type
Layer 1 Troubleshooting Check the outside of the cable Confirm the cable specification with a TDR
Network Connection LED Status Indicators
Link Light (no light, no connection)
Attenuation
Loss of power in a signal as it travels from the sending device to the receiving device Can be measured with a WiFi analyzer Boost the strength on the access point itself Use an antenna with higher gain Move closer to the access point
netstat
Network statistics A TCP/IP troubleshooting utility that displays statistics and the state of current TCP/IP connections. It also displays ports, which can signal whether services are using the correct ports. netstat -a: Show all active connections + UDP/TCP ports which are listening (not just established connections) netstat -b: Show binaries (Windows) netstat -f: Shows FQDN netstat -n: Do not resolve names (numbers only) netstat -o: Shows process ID From Jason Dion test: Netstat (network statistics) is a command-line network utility tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interface and network protocol statistics. It is useful when trying to determine if a workstation is attempting outbound connections due to malware (beaconing activity), or has ports open and listening for inbound connections.
Unresponsive service
No response to an application request Do you have the right port number? Confirm connectivity (ping/traceroute) Try telneting to the port itself
Hardware failure
No response, application doesn't respond Confirm connectivity (without ping, you can't connect) Run a traceroute Check the physical server (lights?)
Crosstalk
Occurs when the signals of one wire affect the signals on an adjacent wire TDR can be used to diagnose it. Near End Crosstalk (NEXT): XT measured at the transmitting end Far End Crosstalk: XT measured at the receiving end Check the cable, maintain the twists, use a better cable (Cat6A has thicker cable diameter)
Incorrect ACL settings
Only certain IPs accessible (or none) Can allow or deny traffic by filtering packets based on criteria (e.g. IP address, port number, other parameters) Confirm with packet captures and TCP/UDP traceroutes
OTDR
Optical time domain reflectometer TDR for copper cables, OTDR for fiber. Detects the location of a fault in a fiber cable by sending light down the fiber-optic cable and measuring the time required for the light to bounce back from the cable fault. The OTDM can then mathematically calculate the location of the fault. Also provides impedance and signal loss information. Good for certifying cable installations Works by sending an electrical pulse which gets reflected if there are any impedance discontinuities. Expensive.
Loopback adapter
Plugs into a port and crosses over the transmit line to the receive line so that outgoing signals can be redirected into the computer for testing.
ping
Sends a message from one computer to another to check whether it is reachable and active. Uses ICMP (Internet Control Message Protocol)
Open/short
Short: Two connections are touching Open: Break in a connection/circuit Often difficult or impossible to repair; better off just replacing the cable TDR can help you find the exact location
Refraction
Signal passes through a medium and exit out at a different angle Changes in air temperature and water vapor affects refraction
Absorption
Signal passes through object and loses signal strength Different frequencies may have different absorption rates Put the antennas on the ceiling
Incorrect time
Some cryptography is time sensitive Kerberos communication uses a time stamp Client won't be able to login Configure NTP on all devices
Interference
Something else is using the same frequencies Some are predictable (microwaves, florescent lights, cordless phone) Some are unpredictable (other companies in a multi-tenant building) Measure with netstat -e or check the performance monitor in windows to track statistics over time
Duplex/speed mismatch
Speed of ethernet connection needs to match on both sides or there will be no connection Duplex needs to be matched, otherwise the speed will suffer
Duplicate IP Addresses
Static address assignments mean you need to be very organized so that you don't accidentally assign duplicates Static IPs and DHCP pools may overlap Multiple DHCP servers may overlap or there may be a rogue DHCP Intermittent connectivity may cause two addresses to "fight" with each other Check your IP addresses ping IP before assigning it
Identify the problem
Step 1 Info gathering Identify symptoms Question users Determine if anything has changed Break problems into smaller pieces
Establish a theory of probable cause
Step 2 Start with the obvious Consider everything (Examine the problem from the top of the OSI model to the bottom) Make a list of all possible causes
Test the theory to determine the cause
Step 3 Confirm the theory Theory didn't work? Re-establish new theory or escalate (e.g. call an expert)
Establish a plan of action to resolve the problem and identify potential effects
Step 4 Build the plan (minimum impact to production, might need to do after hours) Identify potential effects (have a plan B and plan C)
Implement the solution or escalate as necessary
Step 5 Fix the issue (implement during the change control window) Escalate as necessary (may need help from 3rd party)
Verify full system functionality and, if applicable, implement preventive measures
Step 6 Test should be part of your plan Have customer confirm the fix Implement preventive measures to avoid problem in the future
Document findings, actions, and outcomes
Step 7 Build the knowledge base Consider a formal, searchable database
VLAN Mismatch
Switch is configured with the wrong VLAN Link light, but no surfing SSH/connect to the switch to view the VLAN settings Each port should have a VLAN setting (VLAN1 is usually the default)
Incorrect Gateway
Symptoms: Can communicate to local addresses but not ones on a different subnet; no IP communication locally or on different subnets;can communicate to some IPs but not others Check your documentation Check IP settings match documentation Monitor the traffic, examine local broadcasts (maybe you were configured with the wrong VLAN?) Check other devices around you which seem to be working Perform pings on local IPs, default gateway and outside addresses and tracerts
Incorrect Antenna Placement
Symptoms: No or low signal and connectivity. Causes: The position of your antenna can negatively affect overall performance if placed incorrectly. Resolution: Alter the position of your antenna and monitor device performance. Configure different access points with non-overlapping channels
Incorrect pin-out
Test your cables prior to implementation Use a cable tester!
Signal-to-Noise Ratio
The amount of signal present compared to the amount of noise in the environment (the larger the better; 1:1 would be bad)
Incorrect Antenna Type
The antenna must fit the room Omnidirectional useful on a ceiling Connecting two buildings would likely benefit more from a directional antenna The access point may allow you to use different types of antennas
ipconfig
The utility used to display TCP/IP addressing and domain name information in the Windows client operating systems.
Jitter
The variability of latency over time across a network. Confirm available bandwidth Make sure infrastructure is working as expected Apply QoS
Bottlenecks
There's never just one performance metric I/O bus, CPU Speed, Storage Access Speed
Overcapacity
Too many devices on the wireless network 5GHz can help with this, gives you more frequencies to use Lots of data transfers can cause bandwidth saturation
WiFi Analyzer
Tool for checking/diagnosing issues on a wireless network. Some network drivers won't capture wireless info, you'll need a specialized adapter/driver. Lets you view wireless specific info: signal to noise ratio, channel information, etc.
Transceiver Mismatch
Transceiver needs to match the fiber! 850nm fiber needs an 850nm transceiver
iptables
User module used to interact with CLI to enter firewall rules into predefined tables' Some linux distros prefer firewalld
route
View the device's routing table (find out which way the packets go) Windows: route print
Names not resolving
Web browsing doesn't work but ping does. Applications not communicating (they often use names and not IP addresses) Check your IP configuration (DNS correct?) Use nslookup or dig to perform queries against that DNS server If those DNS servers are not responding, try different DNS servers (e.g. Google 8.8.4.4./8.8.8.8 or Quad9 9.9.9.9)
Reflection
Wireless signals can bounce off some surfaces Depends on the surface and frequency Position antennas to mitigate excessive reflection
TX/RX Reverse
Wiring mistake at the cable ends or on the punchdown block Easy to find with a wire map Some network interfaces can actually correct this (Auto-MDIX)
Bandwidth Speed Tester
Works by transferring a file and measuring the throughput. Useful for pre and post change analysis. May want to run speed tests over different times of the day.