5.1: Network Troubleshooting Methodology, 5.2: Hardware & Software Tools, 5.2: Command Line Tools, 5.3: Common Wired Connectivity and Performance Issues, 5.4: Common Wireless Connectivity and Performance Issues, 5.5: Common Network Service Issues

Attenuation

Loss of power in a signal as it travels from the sending device to the receiving device. Decibels are used to measure signal strength.

pathping

A TCP / IP command that provides information about latency and packet loss on a network. Included in Windows NT and later. First phase: Builds a map by running traceroute Second phase: Measures round trip time and packet loss at each hop.

ifconfig

A TCP/IP configuration and management utility used with UNIX and Linux systems.

tcpdump

A command-line protocol analyzer. Administrators use it to capture packets. Available in most unix/linux distros. Included with Mac, available for Windows (WinDump). Dump is written in a standard pcap format (readable by things like Wireshark)

Latency

A delay between the request and the response. Common to use packet captures and protocol analyzers to troubleshoot.

Packet Sniffer

A device or program that monitors network communications and captures data.

Light meter

A device to measure light intensity. Good for validating optical cables.

nslookup

Command-line program in Windows used to determine exactly what information the DNS server is providing about a specific host name. Deprecated (use dig instead)

Duplicate MAC Addresses

Don't see this problem very often May be a misconfiguration, manufacturering error, a man in the middle attack Ping the IP of the device and then check the ARP cache

Multimeter

A measuring instrument for current, AC & DC voltage, and resistance

nmap

A network utility designed to scan a network and create a map. Frequently used as a vulnerability/port scanner. Identifies open ports. Discover OS without logging into a device. Service scan. Includes its own scripting languag (NSE=Nmap Scripting Engine)

Tone generator

A small electronic device that issues a signal on a wire pair. When used in conjunction with a tone locator, it can help locate the termination of a wire pair.

Spectrum Analyzer

A software tool that assesses the characteristics (for example, frequency, amplitude, and the effects of interference) of wireless signals.

Cable tester

A tool for testing if a cable can connect properly end-to-end and to determine if a cable has a short (continuity test). These tools are available for a variety of cable types.

Crimper

A tool used to attach a terminator or connector to the end of a cable. Often includes a wire stripper.

Punchdown Tool

A tool used to punch individual wires from a network cable into their slots to terminate the cable. 66 and 110 blocks most commonly.

Port Scanner

A type of software that searches a network host for open ports (vulnerabilities). Can be used to visually map the network.

arp

Address Resolution Protocol. Determine a MAC address based on an IP address arp -a: View local arp table/cache

Distance Limitations

All signals degrade -Electrical or optical All devices must adhere to a standard Almost always test in advance to measure signal

Blocked TCP/UDP Ports

Applications not working FIrewall or ACL configuration Confirm with a packet capture Run a TCP or UDP-based traceroute tool

Incorrect host-based firewall settings

Applications not working (based on the application in use) Check the host-based firewall settings Take a packet capture from an external device

Untrusted SSL Certificate

Browsers trust certificates from certain CAs Certificate was signed by a CA not on the list If it's an internal server, add your internal CA certificate to the list

Incorrect Netmask

Check your IP configuration, check your documentation Check devices around you

Exhausted DHCP Scope

Clients receive APIPA address (local subnet communication only) Check the DHCP server (add more IP addresses to the pool if possible) IP address management (IPAM) may help; monitor and report on shortages Lower the lease time if you have a lot of transient users

Damaged cables

Copper cables are pretty rugged but not indestructible Can be stepped on or bent Check your physical layer Use a TDR to check within the cable

Expired IP Address

DHCP address should renew before lease expires Check 169.254.*.* for an APIPA assigned address Check the status of your DHCP server

Latency

Delay between transmission and response

tracert (Windows) traceroute (Linux/Mac)

Determine the route a packet takes to a destination. Takes advantage of ICMP Time to Live Exceeded error message (TTL=1 is first router, TTL=2 is second router, etc.) Not all devices will reply with a TTL exceeded message (e.g. firewalls will filter them)

Jitter

Deviation from a predictable data stream

Frequency Mismatch

Devices have to match the access point's available frequencies Verify the client is communicating on the appropriate channel Older standards may slowdown a newer network (802.11b compatibility mode on 802.11g devices)

dig

Domain Information Groper Lookup information from DNS servers. More advanced domain info than nslookup. Not native to any Windows distributions but you can download it.

Channel Overlap

Easier on 5GHz since you have more channels to work with

EMI

Electromagnetic Interference Watch your bend radius, watch your cable ties, don't twist them Avoid power cables, fluorescent lights, fire prevention components Test after installation with a TDR

Security Type Mismatch

Encryption on wireless is important Needs to match between client and access point Migrate all WEP and WPA to WPA2!

Wrong SSID

Every access point has at least one Service Set Identifier (SSID) Confirm the correct SSID settings

Protocol Analyzer

Hardware or software that captures packets to decode and analyze their contents. Good for viewing traffic patterns and doing big data analytics.

Rogue DHCP Server

IP addresses assigned by a non-authorized server Client is assigned an invalid or duplicate address Disable rougue DHCP communication Enable DHCP snooping on your switch Make sure authorized DHCP servers are on Active Directory Release IP addresses and renew with the correct DHCP servers

Power Levels

Increase the power level on an access point to increase its range.

Wrong Passphrase

Is not connected, check the authentication Shared passphrase common in SOHO, not enteprise 802.1X is commonly used in enterprise settings; make sure client is configured to run 802.1X

Incorrect cable type

Layer 1 Troubleshooting Check the outside of the cable Confirm the cable specification with a TDR

Network Connection LED Status Indicators

Link Light (no light, no connection)

Attenuation

Loss of power in a signal as it travels from the sending device to the receiving device Can be measured with a WiFi analyzer Boost the strength on the access point itself Use an antenna with higher gain Move closer to the access point

netstat

Network statistics A TCP/IP troubleshooting utility that displays statistics and the state of current TCP/IP connections. It also displays ports, which can signal whether services are using the correct ports. netstat -a: Show all active connections + UDP/TCP ports which are listening (not just established connections) netstat -b: Show binaries (Windows) netstat -f: Shows FQDN netstat -n: Do not resolve names (numbers only) netstat -o: Shows process ID From Jason Dion test: Netstat (network statistics) is a command-line network utility tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interface and network protocol statistics. It is useful when trying to determine if a workstation is attempting outbound connections due to malware (beaconing activity), or has ports open and listening for inbound connections.

Unresponsive service

No response to an application request Do you have the right port number? Confirm connectivity (ping/traceroute) Try telneting to the port itself

Hardware failure

No response, application doesn't respond Confirm connectivity (without ping, you can't connect) Run a traceroute Check the physical server (lights?)

Crosstalk

Occurs when the signals of one wire affect the signals on an adjacent wire TDR can be used to diagnose it. Near End Crosstalk (NEXT): XT measured at the transmitting end Far End Crosstalk: XT measured at the receiving end Check the cable, maintain the twists, use a better cable (Cat6A has thicker cable diameter)

Incorrect ACL settings

Only certain IPs accessible (or none) Can allow or deny traffic by filtering packets based on criteria (e.g. IP address, port number, other parameters) Confirm with packet captures and TCP/UDP traceroutes

OTDR

Optical time domain reflectometer TDR for copper cables, OTDR for fiber. Detects the location of a fault in a fiber cable by sending light down the fiber-optic cable and measuring the time required for the light to bounce back from the cable fault. The OTDM can then mathematically calculate the location of the fault. Also provides impedance and signal loss information. Good for certifying cable installations Works by sending an electrical pulse which gets reflected if there are any impedance discontinuities. Expensive.

Loopback adapter

Plugs into a port and crosses over the transmit line to the receive line so that outgoing signals can be redirected into the computer for testing.

ping

Sends a message from one computer to another to check whether it is reachable and active. Uses ICMP (Internet Control Message Protocol)

Open/short

Short: Two connections are touching Open: Break in a connection/circuit Often difficult or impossible to repair; better off just replacing the cable TDR can help you find the exact location

Refraction

Signal passes through a medium and exit out at a different angle Changes in air temperature and water vapor affects refraction

Absorption

Signal passes through object and loses signal strength Different frequencies may have different absorption rates Put the antennas on the ceiling

Incorrect time

Some cryptography is time sensitive Kerberos communication uses a time stamp Client won't be able to login Configure NTP on all devices

Interference

Something else is using the same frequencies Some are predictable (microwaves, florescent lights, cordless phone) Some are unpredictable (other companies in a multi-tenant building) Measure with netstat -e or check the performance monitor in windows to track statistics over time

Duplex/speed mismatch

Speed of ethernet connection needs to match on both sides or there will be no connection Duplex needs to be matched, otherwise the speed will suffer

Duplicate IP Addresses

Static address assignments mean you need to be very organized so that you don't accidentally assign duplicates Static IPs and DHCP pools may overlap Multiple DHCP servers may overlap or there may be a rogue DHCP Intermittent connectivity may cause two addresses to "fight" with each other Check your IP addresses ping IP before assigning it

Identify the problem

Step 1 Info gathering Identify symptoms Question users Determine if anything has changed Break problems into smaller pieces

Establish a theory of probable cause

Step 2 Start with the obvious Consider everything (Examine the problem from the top of the OSI model to the bottom) Make a list of all possible causes

Test the theory to determine the cause

Step 3 Confirm the theory Theory didn't work? Re-establish new theory or escalate (e.g. call an expert)

Establish a plan of action to resolve the problem and identify potential effects

Step 4 Build the plan (minimum impact to production, might need to do after hours) Identify potential effects (have a plan B and plan C)

Implement the solution or escalate as necessary

Step 5 Fix the issue (implement during the change control window) Escalate as necessary (may need help from 3rd party)

Verify full system functionality and, if applicable, implement preventive measures

Step 6 Test should be part of your plan Have customer confirm the fix Implement preventive measures to avoid problem in the future

Document findings, actions, and outcomes

Step 7 Build the knowledge base Consider a formal, searchable database

VLAN Mismatch

Switch is configured with the wrong VLAN Link light, but no surfing SSH/connect to the switch to view the VLAN settings Each port should have a VLAN setting (VLAN1 is usually the default)

Incorrect Gateway

Symptoms: Can communicate to local addresses but not ones on a different subnet; no IP communication locally or on different subnets;can communicate to some IPs but not others Check your documentation Check IP settings match documentation Monitor the traffic, examine local broadcasts (maybe you were configured with the wrong VLAN?) Check other devices around you which seem to be working Perform pings on local IPs, default gateway and outside addresses and tracerts

Incorrect Antenna Placement

Symptoms: No or low signal and connectivity. Causes: The position of your antenna can negatively affect overall performance if placed incorrectly. Resolution: Alter the position of your antenna and monitor device performance. Configure different access points with non-overlapping channels

Incorrect pin-out

Test your cables prior to implementation Use a cable tester!

Signal-to-Noise Ratio

The amount of signal present compared to the amount of noise in the environment (the larger the better; 1:1 would be bad)

Incorrect Antenna Type

The antenna must fit the room Omnidirectional useful on a ceiling Connecting two buildings would likely benefit more from a directional antenna The access point may allow you to use different types of antennas

ipconfig

The utility used to display TCP/IP addressing and domain name information in the Windows client operating systems.

Jitter

The variability of latency over time across a network. Confirm available bandwidth Make sure infrastructure is working as expected Apply QoS

Bottlenecks

There's never just one performance metric I/O bus, CPU Speed, Storage Access Speed

Overcapacity

Too many devices on the wireless network 5GHz can help with this, gives you more frequencies to use Lots of data transfers can cause bandwidth saturation

WiFi Analyzer

Tool for checking/diagnosing issues on a wireless network. Some network drivers won't capture wireless info, you'll need a specialized adapter/driver. Lets you view wireless specific info: signal to noise ratio, channel information, etc.

Transceiver Mismatch

Transceiver needs to match the fiber! 850nm fiber needs an 850nm transceiver

iptables

User module used to interact with CLI to enter firewall rules into predefined tables' Some linux distros prefer firewalld

route

View the device's routing table (find out which way the packets go) Windows: route print

Names not resolving

Web browsing doesn't work but ping does. Applications not communicating (they often use names and not IP addresses) Check your IP configuration (DNS correct?) Use nslookup or dig to perform queries against that DNS server If those DNS servers are not responding, try different DNS servers (e.g. Google 8.8.4.4./8.8.8.8 or Quad9 9.9.9.9)

Reflection

Wireless signals can bounce off some surfaces Depends on the surface and frequency Position antennas to mitigate excessive reflection

TX/RX Reverse

Wiring mistake at the cable ends or on the punchdown block Easy to find with a wire map Some network interfaces can actually correct this (Auto-MDIX)

Bandwidth Speed Tester

Works by transferring a file and measuring the throughput. Useful for pre and post change analysis. May want to run speed tests over different times of the day.