Network Troubleshooting

Identifying and Solving Problems on

the Network

Reference: Network Troubleshooting

Tools, Joseph D. Sloan, O'Reilly,
August 2001
 Focus: Basics and Standard Tools
• Solving network problems depends a
lot on your understanding
• Simple tools can tell you what you
need to know
• Example: ping is incredibly useful!

Systems and Network Mana Network Troubleshooting 2

gement
 Troubleshooting
• Avoid it by:
– redundancy
– documentation
– training
• Try quick fixes first
– simple problems often have big effects:
– is the power on?
– is the network cable plugged into the right socket? Is LED flashing?
– has anything changed recently?
• Change only one thing at a time
– test thoroughly after the change
• Be familiar with the system
– maintain documentation
• Be familiar with your tools
– before trouble strikes

Systems and Network Mana Network Troubleshooting 3

gement
 Troubleshooting: Learn as you go
• Study and be familiar with the normal
behaviour of your network
• Monitoring tools can tell you when things
are wrong
– if you know what things look like when they are
right
• Using tools such as Ethereal can help you
understand
– your network, and
– TCP/IP — better
Systems and Network Mana Network Troubleshooting 4
gement
 Documentation
• Maintain an inventory of equipment and software
– a list mapping MAC addresses to machines can be very helpful
• Maintain a change log for each major system, recording:
– each significant change
– each problem with the system
– each entry dated, with name of person who made the entry
• Two categories of documentation:
– Configuration information
• describes the system
• use system tools to obtain a snapshot, e.g., sysreport in Red Hat Linux
– Procedural information
• How to do things
• use tools that automatically document what you are doing, e.g., script

Systems and Network Mana Network Troubleshooting 5

gement
 Documentation Tools
• Use script:
$ script ~/logs/logfile-$(date +%F-%R).log
– starts a new shell
– all you type, all output goes into the file
– Add comments with # I tried this...
• Use tee:
$ arp –a | tee outfile
• Use sudo: all commands are recorded in /var/log/secure
• Use plod from
http://bullwinkle.deer-run.com/~hal/plod/
– lets you record a worksheet easily
– Perl, so fine on any platform

Systems and Network Mana Network Troubleshooting 6

gement
 Purchasing Equipment
• Better to:
• spend enough for the short term (one or two years) or
• “invest for the long term?”
• Moore’s Law: exponential growth in “bang for the buck”
• Maintenance costs more for older equipment
• Count all the costs
• Conclusion: often (but not always), getting cheaper
equipment to cover needs for the next two years will save
money
• Buying excess capacity can waste a lot of money

Systems and Network Mana Network Troubleshooting 7

gement
 Host Network Configuration tools
• ps — information about processes
• top — dynamic information about processes
• netstat — show connections and services, routing
• lsof — list open files
• ifconfig — shows and changes network
interfaces
• route — shows, changes routing table
• ip — show, change, set network configuration
• arp — shows MAC addresses
• nmap — portscanner: shows open ports

Systems and Network Mana Network Troubleshooting 8

gement
 Using ps to See If Server Running
• Is the network service running on the
server?
• Is the web server running?
• ps aux | grep httpd
• Is the DHCP server running?
• ps aux | grep dhcpd
• Is the directory server running?
• ps aux | grep slapd
• Windows: use the task manager
Systems and Network Mana Network Troubleshooting 9
gement
 Using top to See Resource Hogs
• The program top shows:
– load average (the average number of
processes that are ready to run, but for
which no CPU is available)
• a load average of 4 or more is “quite high”
– processes that use the most resources

Systems and Network Mana Network Troubleshooting 10

gement
 Using netstat –tua to See
Network Connections
• netstat –tua shows all network
connections, including those listening
• netstat –tu shows only connections that
are established
• netstat –i is like ifconfig, shows info
and stats about each interface
• netstat –nr shows the routing table, like
route –n
• Windows provides netstat also.

Systems and Network Mana Network Troubleshooting 11

gement
 lsof: List Open Files
• An amazingly useful tool
• Available for almost any Unix system
• lsof –i shows output to Internet and X.25 files,
but won’t show connections that have terminated
• lsof –[email protected] will show
only connections to that machine
• Can monitor progress of an FTP transfer, many,
many other applications
• See manpage, FAQ and quick start guide.
• Apparently, no equivalent tool available on
Windows.
Systems and Network Mana Network Troubleshooting 12
gement
 ifconfig
• ifconfig eth0 — show stats on network
interface eth0
• sudo ifconfig lo 127.0.0.1 — configure the
loopback interface, start it up
• sudo ifconfig eth0 172.19.233.5 netmask
255.255.255.0 — configure eth0 with IP
address 172.19.233.5/24
• ifconfig — show all configured network
interfaces
• ifconfig –a — show all interfaces, including
those not configured yet.

Systems and Network Mana Network Troubleshooting 13

gement
 route
• route –n — print routing table
• route add 127.0.0.1 — add a route to localhost;
– should have been done automatically when created device with
ifconfig
• route add –net 172.19.233.0 — add a route to the
eth0 configured on previous slide
– should have been done automatically by ifconfig
• route add 172.19.64.0 gw 172.19.233.254 — add a
static route to network 172.19.64.0 through router
172.19.233.254
• route add default gw 172.19.233.253 — add a
default route to 172.19.233.253 through eth0

Systems and Network Mana Network Troubleshooting 14

gement
 Connectivity Testing: Cabling
• Label cables clearly at each end
• Cable testers
– ensure wired correctly, check:
– attenuation
– length — is it too long?
• 100BaseT: less than 100m
• Is the activity light on the interface
blinking?

Systems and Network Mana Network Troubleshooting 15

gement
 Software tools: ping
• Most useful check of connectivity
• Universal
• If ping hostname, includes a rough check of DNS
• Sends an ICMP (Internet Control Message
Protocol) ECHO_REQUEST
• Waits for an ICMP ECHO_REPLY
• Most pings can display round trip time
• Most pings can allow setting size of packet
• Can use to make a crude measurement of
throughput

Systems and Network Mana Network Troubleshooting 16

gement
ping: Roughly Estimating Throughput
• Example:
• ping with packet size = 100 bytes, round-trip time =
30ms
• ping with packet size = 1100 bytes, round-trip time =
60ms
• So takes 30ms extra (15ms one way) to send
additional 1000 bytes, or 8000 bits
• Throughput is roughly 8000 bits per 15ms, or about
540,000 bits per second
• A very crude measurement: no account for other
traffic, treats all links on path, there and back, as
one.
Systems and Network Mana Network Troubleshooting 17
gement
ping: Roughly Estimating Throughput
• This can be expressed as a simple
formula:
Pl  Ps
TP  16  bits per second, where
tl  t s
Pl  size of large packet
Ps  size of small packet
tl  round  trip ping time for large packet
ts  round - trip ping time for small packet
Systems and Network Mana Network Troubleshooting 18
gement
 What ping Result is Good, Bad?
• A steady stream of consistent replies indicates
probably okay
• Usually first reply takes longer due to ARP lookups
at each router
– After that, ARP results are cached
• ICMP error messages can help understand results:
– Destination Network Unreachable indicates the host
doing ping cannot reach the network
– Destination Host Unreachable may come from routers
further away

Systems and Network Mana Network Troubleshooting 19

gement
 How to Use ping?
• Ensure local host networking is enabled
first: ping localhost, local IP address
• ping a known host on local network
• ping local and remote interfaces on router
• ping by IP as well as by hostname if
hostname ping fails
– confirm DNS with dig (or nslookup) — see later
• Ping from more than one host

Systems and Network Mana Network Troubleshooting 20

gement
 fping: flood ping
• Designed to test a large number of hosts
• more efficient than ping
• Used extensively by monitoring software
such as mon:
http://www.kernel.org/software/mon/,
nagios: http://www.nagios.org/
• take care not to flood to much!
• RPMs are available; I have built one and put
on ictlab under ~ftp/pub/redhat/contrib

Systems and Network Mana Network Troubleshooting 21

gement
 arping: uses ARP requests
• Limited to local network
• Can work with MAC or IP addresses
• use to probe for ARP entries in
router (very useful!)
• packet filtering
– can block ICMP pings, but
– won't block ARP requests

Systems and Network Mana Network Troubleshooting 22

gement
 Path Discovery: traceroute
• Sends UDP packets
– (Microsoft tracert sends ICMP packets)
• increments Time to Live (TTL) in IP
packet header
• Sends three packets at each TTL
• records round trip time for each
• increases TTL until enough to reach
destination
Systems and Network Mana Network Troubleshooting 23
gement
 traceroute: How it Works
• As IP packets pass through each router,
TTL in IP header is decremented
• Packet is discarded when TTL decrements
to 0
• ROUTER sends ICMP TIME_EXCEEDED
message back to traceroute host
• When UPD packet reaches destination,
gets ICMP PORT_UNREACHABLE, since
uses an unused high UDP port

Systems and Network Mana Network Troubleshooting 24

gement
 traceroute Limitations
• Each router has a number of IP addresses
• but traceroute only shows the one it used
• get different addresses when run
traceroute from other end
• sometimes route is asymmetric
• router may be configured to not send ICMP
TIME_EXCEEDED messages
– get stars: * instead of round-trip time in
traceroute output

Systems and Network Mana Network Troubleshooting 25

gement
 Performance Measurements: delay
• Three sources of delay:
• transmission delay — time to put signal
onto cable or media
– depends on transmission rate and size of frame
• propagation delay — time for signal to
travel across the media
– determined by type of media and distance
• queuing delay — time spent waiting for
retransmission in a router

Systems and Network Mana Network Troubleshooting 26

gement
 Performance Measurements 2
• bandwidth — the transmission rate through
the link
– relates to transmission time
• throughput — amount of data that can be
sent over link in given time
– relates to all causes of delay
– is not the same as bandwidth
• Other measurements needed
– I.e., for quality of service for multimedia
Systems and Network Mana Network Troubleshooting 27
gement
Throughput: Measuring with ping 1
• Measure throughput between two remote hosts:
may use tools like ping
• ping two locations with two packet sizes (4 pings
altogether, minimum)
• Example:
Address RTT 100 bytes RTT 1100 bytes
205.153.61.1 1.380 ms 5.805 ms
205.153.60.2 4.985 ms 12.823 ms
165.166.36.17 8.621 ms 26.713 ms

Systems and Network Mana Network Troubleshooting 28

gement
Throughput: Measuring with ping 2
Address RTT 100 bytes RTT 1100 bytes
205.153.61.1 1.380 ms 5.805 ms
205.153.60.2 4.985 ms 12.823 ms
165.166.36.17 8.621 ms 26.713 ms

• Time difference / 2 (round trip time (RTT) -> one way)

• Divide by size difference in bits: 8000
• Multiply by 1000 (ms -> seconds)
• Convert bps to Mbps

Near link Far Link Time difference Est. Throughput

205.153.61.1 205.153.60.2 3.413 ms 4.69 Mbps
205.153.60.2 165.166.36.17 10.254 ms 1.56 Mbps

Systems and Network Mana Network Troubleshooting 29

gement
Throughput: Measuring with ping 3
TP  16  ( Pl  Ps ) /(t2l  t2 s  t1l  t1s )
Pl  larger packet size
Ps  smaller packet size
t1l  ping time for larger packet to near link
t1s  ping time for smaller packet to near link
t2l  ping time for larger packet to far link
t2 s  ping time for smaller packet to far link
Systems and Network Mana Network Troubleshooting 30
gement
Throughput: Measuring with ping 4
TP  16  ( Pl  Ps ) /(t 2l  t 2 s  t1l  t1s )
Pl  1100
Ps  100
3
t1l  5.805 10
3
t1s  1.380 10
3
t 2l  12.823  10
3
t 2 s  4.985  10
Systems and Network Mana Network Troubleshooting 31
gement
Throughput: Measuring with ping 5
• Completing calculation for throughput between 205.153.61.1 and 205.153.61.2:

Pl  1100
Ps  100
The time difference :
t 2l  t 2 s  t1l  t1s  (12.823  4.985  5.805  1.380)  103
 3.413103
so throughput is :
TP  16  (1100  100) /((12.823  4.985  5.805  1.380) 103
 4,687,958
 4.69 Mbps
Systems and Network Mana Network Troubleshooting 32
gement
 Path Performance: Other tools
• Could use a tool like pathchar, bing, clink, pchar, or
tmetric that performs this calculation for you
• Use http://www.google.com to locate these tools
• pathchar is only available in binary form
• Others in source form, need compile with
commands something like this:
$ cd bing-1.1.3
$ make
$ sudo make install

Systems and Network Mana Network Troubleshooting 33

gement
 Path measurement with pathchar
$ sudo ./pathchar sina.com.hk
pathchar to sina.com.hk (202.85.139.140)
can't find path mtu - using 1500 bytes.
doing 32 probes at each of 45 sizes (64 to 1500 by 32)
0 localhost (127.0.0.1)
| 106 Mb/s, 293 us (698 us), +q 1.18 ms (15.7 KB)
1 172.19.35.246 (172.19.35.246)
| 28 Mb/s, 488 us (2.10 ms)
2 192.168.83.2 (192.168.83.2)
3 * 1 448 798 2
| 20 Mb/s, 273 us (3.25 ms)
4 cw7204.vtc.edu.hk (202.40.210.220)
| 6.8 Mb/s, 521 us (6.04 ms)
5 210.176.123.37 (210.176.123.37)
| 52 Mb/s, 20 us (6.31 ms)
6 210.87.254.61 (210.87.254.61)
| 136 Mb/s, 116 us (6.63 ms)
7 g5-0-0.wttbr01.imsbiz.com (210.87.254.129)
| 33 Mb/s, 0.94 ms (8.88 ms), +q 1.48 ms (6.10 KB) *6
8 iadvantage3-RGE.hkix.net (202.40.161.172)
| 164 Mb/s, 45 us (9.04 ms), +q 1.74 ms (35.6 KB) *6
9 v005-m02.hk01.iadvantage.net (202.85.129.53)
| ?? b/s, -66 us (8.88 ms)
10 202.85.129.136 (202.85.129.136)
| ?? b/s, 459 us (9.79 ms)
11 202.85.139.11 (202.85.139.11)
11 hops, rtt 6.18 ms (9.79 ms), bottleneck 6.8 Mb/s, pipe 9361 bytes

Systems and Network Mana Network Troubleshooting 34

gement
 Path Performance: measuring
• May use ftp to transfer a large file,
measure time
– tests whole path
– problem: affected by disk I/O, xinetd
• Use ttcp, not affected by disk I/O
• Consists of a client and server
• Need have installed at both ends
• Part of Red Hat Linux, Cisco IOS
Systems and Network Mana Network Troubleshooting 35
gement
 Example of use of ttcp
• First, start receiver on ictlab:
$ ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 172.19.32.30
ttcp-r: 16777216 bytes in 1.45 real seconds = 11285.88 KB/sec +++
ttcp-r: 9704 I/O calls, msec/call = 0.15, calls/sec = 6684.46
ttcp-r: 0.0user 0.2sys 0:01real 14% 0i+0d 0maxrss 0+2pf 0+0csw
• Second, start transmitter on nickpc:
$ ttcp -t -s ictlab
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp -> ictlab
ttcp-t: socket
ttcp-t: connect
ttcp-t: 16777216 bytes in 1.45 real seconds = 11335.64 KB/sec +++
ttcp-t: 2048 I/O calls, msec/call = 0.72, calls/sec = 1416.95
ttcp-t: 0.0user 0.0sys 0:01real 4% 0i+0d 0maxrss 0+2pf 0+0csw

Systems and Network Mana Network Troubleshooting 36

gement
 The ip program, iproute
• The ip program in the iproute package
provides complete control over TCP/IP
networking in a Linux system
• Provides more networking control facilities
than other TCP/IP implementations
• Supports tunneling in many forms
• iproute documentation is in two manuals,
one for ip routing, the other for tunneling

Systems and Network Mana Network Troubleshooting 37

gement
 iproute and iptables
• Between these software packages, you can:
– throttle bandwidth for certain computers
– throttle bandwidth to certain computers
– fairly share bandwidth
– protect your network from DoS attacks
– protect Internet from your customers
– multiplex many servers into one, for load balancing or for high
availability
– restrict access to your computers
– limit access of your users to other hosts
– do routing based on user id, MAC address, source IP, port, type
of service, time of day or content
• See the Linux Advanced Routing and Traffic Control HOWTO
at http://tldp.org for details

Systems and Network Mana Network Troubleshooting 38

gement
 Traffic Measurements: netstat -i
• The netstat program can show statistics about network
interfaces
• Linux netstat shows lost packets in three categories:
– errors,
– drops (queue full: shouldn’t happen!)
– overruns (last data overwritten by new data before old data
was read: shouldn’t happen!)
– drops and overruns indicate faulty flow control — bad!
• These values are cumulative (since interface was up)
• Could put a load on interface to see current condition, with
ping –l, to send large number of packets to destination
• See the difference in values

Systems and Network Mana Network Troubleshooting 39

gement
 Measuring Traffic: netstat -i
• Here we run netstat –i on ictlab:
$ netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 407027830 0 0 0 1603191764 0 0 3 BMRU
lo 16436 0 2858402 0 0 0 2858402 0 0 0 LRU
• Notice that of the 1.6 billion bytes transmitted, there were 3
overuns.
• Next, blast the path you want to test with packets using ping –l or
the spray program, and measure again.

Systems and Network Mana Network Troubleshooting 40

gement
 Traffic measurements: ifconfig, ip
• ifconfig and ip give more information than netstat –i:
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:00:E2:35:AF:EE
inet addr:172.19.64.52 Bcast:172.19.127.255 Mask:255.255.192.0
IPX/Ethernet 802.2 addr:33001601:0000E235AFEE
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:407579600 errors:0 dropped:0 overruns:0 frame:0
TX packets:1605655688 errors:0 dropped:0 overruns:3 carrier:0
collisions:0 txqueuelen:100
RX bytes:3055300191 (2913.7 Mb) TX bytes:2048217058 (1953.3 Mb)
Interrupt:18 Base address:0xd000
$ ip -s link list eth0
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:00:e2:35:af:ee brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
3058362227 407610495 0 0 0 0
TX: bytes packets errors dropped carrier collsns
2140511920 1605768150 0 0 0 0

Systems and Network Mana Network Troubleshooting 41

gement
 Getting more info using ip
• The –s (-statistics) option to ip provides statistics. Adding a
second gives you even more:
$ ip -s -s link list eth0
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:00:e2:35:af:ee brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
3070792102 407726727 0 0 0 0
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
2445799644 1606151878 0 0 0 0
TX errors: aborted fifo window heartbeat
0 3 0 0

Systems and Network Mana Network Troubleshooting 42

gement
 Quick Guide to using ip: set up
interface
• Here we set up a network interface and give
it the IP address 192.168.0.1/24:
$ ip link set dev eth1 up
$ ip addr add 192.168.0.1/24 brd + dev eth1
• Two important points:
– If you do not specify the netmask, a netmask
of /32 is assumed
– brd + means obtain broadcast address by setting
the host bits

Systems and Network Mana Network Troubleshooting 43

gement
 Quick Guide to using ip: set up
routes
$ ip route add default dev eth1 via 192.168.0.254
$ ip route add 192.168.1.0/24 via 192.168.0.10
• The last adds a static route to another network
• the first adds the default route.
• You can omit the device if the network can be
reached through a particular interface without
any ambiguity
– I.e., ip is smart enough to figure out which network
device to use, though specifying it doesn’t hurt.

Systems and Network Mana Network Troubleshooting 44

gement
Packet Capture

tcpdump, Ethereal, ntop

 What is Packet Capture?
• Real time collection of data as it
travels over networks
• Tools called:
– packet sniffers
– packet analysers
– protocol analysers, and sometimes even
– traffic monitors

Systems and Network Mana Network Troubleshooting 46

gement
 When Packet Capture?
• Most powerful technique
• When need to see what client and
server are actually saying to each
other
• When need to analyse type of traffic
on network
• Requires understanding of network
protocols to use effectively

Systems and Network Mana Network Troubleshooting 47

gement
 Warning: Don’t Get Sacked!
• Be sure that your boss agrees with you
capturing packets on your company’s network
• People have been sacked for doing this
without permission!
• Do not invade the privacy of others
• Capturing passwords with insecure protocols
such as telnet, ftp, http (that is not
encrypted with TLS) is very easy
– DON’T DO IT!

Systems and Network Mana Network Troubleshooting 48

gement
 tcpdump
• Available everywhere
• Windows: http://windump.polito.it/
• Syntax also used by other programs
(such as Ethereal)
• Often it is the only tool available, so
good to know
• Works by putting network interface
into promiscuous mode
Systems and Network Mana Network Troubleshooting 49
gement
 How to use tcpdump
• Can just type its name (as root):
$ sudo tcpdump
• ...but get a huge amount of data!
• Can restrict the data collected using
a filter
• A filter may select addresses,
protocols, port numbers,...

Systems and Network Mana Network Troubleshooting 50

gement
 tcpdump: some options
• -c n — capture a count of n packets then stop
• -w file — write raw data to file.
– Very useful — can filter and analyse this later with tcpdump, ethereal
or other tools
– but you cannot see what you are capturing till later!
• -i interface — collect from interface instead of lowest
numbered network interface
• -s bytes — collect no more than bytes of data from each packet
instead of default 68 bytes
• -e — show link level info, e.g., Ethernet addresses
• -x — gives a hexadecimal dump of packets
– excluding link level data
• -X — display Ascii as well as hexadecimal if have –x option too
• Many more options: man tcpdump

Systems and Network Mana Network Troubleshooting 51

gement
 tcpdump Filters: host and port
• Show all network traffic to and from
192.168.0.1:
tcpdump host 192.168.0.1
• Show packets to 192.168.0.1:
tcpdump dst 192.168.0.1
• Show packets to port 68 on
192.168.0.1:
tcpdump dst 192.168.0.1 and port 68

Systems and Network Mana Network Troubleshooting 52

gement
 tcpdump filters: networks
• Capture traffic to or from
205.153.60/24:
tcpdump net 172.19.64/18
• can specify network as source or
destination:
tcpdump src net 205.153.60/24
tcpdump dst net 172.19.64/18

Systems and Network Mana Network Troubleshooting 53

gement
 tcpdump filters: protocol
• tcpdump ip
• tcpdump tcp
• tcpdump ip proto ospf
• This will catch DNS name lookups, but
not zone transfers (which use tcp):
• tcpdump udp port 53

Systems and Network Mana Network Troubleshooting 54

gement
 tcpdump filters: combining
• This will not work as you might expect:
• tcpdump host ictlab and udp or
arp
• Instead, need group with parentheses, and
quote:
• tcpdump “host ictlab and (udp or arp)”
• many more ways of filtering: man tcpdump

Systems and Network Mana Network Troubleshooting 55

gement
 IP Header Bits

12

31
16

20

24

28
0

8
Version IHL Type of Service Total Length
1

MF
DF
Identification Fragmentation Offset
2

Time to Live Protocol Header Checksum

3
Words

Source Address
4

Destination Address
5

Options (0 to 40 bytes) Padding

5-16
Your data starts here

Systems and Network Mana Network Troubleshooting 56

gement
 TCP Header Bits

28
12

16

20

24

31
0

8
Source Port Destination Port
1

Sequence Number
2

Acknowledgement Number
3
Words

header
URG
ACK

SYN
PSH
RST

FIN
length Reserved Window
4

Checksum Urgent Pointer

5

Options (0 to 40 bytes) Padding

5-15
Your data starts here

Systems and Network Mana Network Troubleshooting 57

gement
 UDP Header
Bits

31
16
0

Source Port Destination Port

Length Checksum

Your data starts here

Systems and Network Mana Network Troubleshooting 58

gement
 Writing data to a file
sudo tcpdump -c 1000 -w ~/tmp/tcpdump.pcap
tcpdump: listening on eth0
1014 packets received by filter
0 packets dropped by kernel

Systems and Network Mana Network Troubleshooting 59

gement
 Reading a dumped file
$ tcpdump -nr ~/tmp/tcpdump.pcap arp
22:32:41.751452 arp who-has 172.19.127.254 tell 172.19.127.29
22:32:41.863173 arp who-has 172.19.64.52 tell 172.19.64.63
22:32:41.863198 arp reply 172.19.64.52 is-at 0:0:e2:35:af:ee
22:32:42.082584 arp who-has 172.19.65.16 tell 172.19.125.229
22:32:43.113655 arp who-has 172.19.123.211 tell 172.19.65.2
22:32:44.635149 arp who-has 172.19.65.16 tell 172.19.127.106
22:32:44.874117 arp who-has 172.19.65.6 tell 172.19.126.174
22:32:45.147178 arp who-has 172.19.65.16 tell 172.19.126.240
22:32:45.209507 arp who-has 172.19.127.254 tell 172.19.125.127
22:32:45.212484 arp who-has 172.19.127.175 tell 172.19.125.127
22:32:45.239445 arp who-has 172.19.127.254 tell 172.19.125.212
22:32:45.455863 arp who-has 172.19.65.16 tell 172.19.126.194
22:32:45.540507 arp who-has 172.19.126.50 (44:30:54:59:43:4d)
tell 172.19.65.10
22:32:45.562004 arp who-has 172.19.126.50 tell 172.19.65.2

Systems and Network Mana Network Troubleshooting 60

gement
 HTTP
tcpdump -nr ~/tmp/tcpdump.pcap port http
22:43:32.633636 192.168.25.9.14075 > 172.19.64.52.http: S
1015952778:1015952778(0) win 6144 <mss 1460> (DF)
22:43:32.633693 172.19.64.52.http > 192.168.25.9.14075: S
1929920485:1929920485(0) ack 1015952779 win 5840 <mss 1460>
(DF)
22:43:32.635828 192.168.25.9.14075 > 172.19.64.52.http: P
1:590(589) ack 1 win 6144 (DF)
22:43:32.635906 172.19.64.52.http > 192.168.25.9.14075: . ack
590 win 6479 (DF)
22:43:32.636758 172.19.64.52.http > 192.168.25.9.14075: P
1:217(216) ack 590 win 6479 (DF)
22:43:32.636982 172.19.64.52.http > 192.168.25.9.14075: F
217:217(0) ack 590 win 6479 (DF)
22:43:32.639080 192.168.25.9.14075 > 172.19.64.52.http: R
590:590(0) ack 217 win 0 (DF)

Systems and Network Mana Network Troubleshooting 61

gement
 tcpdump: When reading TCP
• format:
– src > dst: flags data-seqno ack window urgent
options
• Flags are some combination of S (SYN), F (FIN), P
(PUSH) or R (RST) or a single '.' (no flags).
• The first time tcpdump sees a tcp 'conversation',
it prints the sequence number from the packet.
• On subsequent packets of the conversation, the
difference between the current packet's sequence
number and this initial sequence number is printed.

Systems and Network Mana Network Troubleshooting 62

gement
 Window
• win nnnn specifies data window the
sending host will accept in future
packets
– I.e., the maximum number of bytes
• TCP flow-control:
– host reduces this number if congested
or overloaded
– will sometimes set to 0 to temporarily
halt incoming traffic in this connection
Systems and Network Mana Network Troubleshooting 63
gement
 Ethereal

King of the Packet Analysers!

Available for Linux, Unix, Windows
 Ethereal
• Ethereal can read data captured by
tcpdump, e.g.,
$ ethereal –r tcpdump.pcap
• or File -> Open
• Can capture data itself
• Uses same filter language as tcpdump

Systems and Network Mana Network Troubleshooting 65

gement
Systems and Network Mana Network Troubleshooting 66
gement
Systems and Network Mana Network Troubleshooting 67
gement
 You can expand any protocol:
• If we click on the + next to Bootstrap
Protocol, we can see the details of
the DHCP Request:

Systems and Network Mana Network Troubleshooting 68

gement
Systems and Network Mana Network Troubleshooting 69
gement
 Display Filters
• Note the box at the bottom of Ethereal for display
filters
• Select only some of the packets captured for
display
• see man ethereal and search for DISPLAY FILTER
SYNTAX
• Different syntax than the syntax for capture
filters
• Example:
ip.src==172.19.64.52 and ip.dest==172.19.64.57
Systems and Network Mana Network Troubleshooting 70
gement
 Tools -> Follow TCP Stream
• Can view the contents of an entire
TCP stream conversation, in ASCII or
in hexadecimal.
• Be careful not to invade your
customers’ privacy.
• Can use to check if a communications
stream is really encrypted

Systems and Network Mana Network Troubleshooting 71

gement
 Ntop: monitoring data at a point
• The ntop program
– listens on a network interface
– puts an Ethernet interface into promiscuous mode
and
– displays statistics through a web interface
• Shows:
– percentages of protocols,
– which machines generate most traffic
– which traffic is purely local, which traffic comes
from outside, which traffic goes from inside to
outside of network
Systems and Network Mana Network Troubleshooting 72
gement
 Ntop RPM
• I have made an RPM package of ntop
– it’s the best one available, or at least it was
when I made it :-)
• Can get from
/home/nfs/redhat/contrib/ntop-2.1.51-
20021031nu2.i386.rpm
– source rpm is there too
• Or search for it on http://rpmfind.net/
• Note that you will be prompted for a
password when you install it.
Systems and Network Mana Network Troubleshooting 73
gement
 Switched Networks
• Problem: a switched network is really a point-to-
point network
• You cannot normally capture the unicast traffic
from other hosts on a single switch port
• Solution: many switches support port monitoring,
where one port can monitor all traffic on a
specified VLAN
• Example: Cisco 3500XL switches provide the port
monitor command:
• port monitor vlan VLAN1

Systems and Network Mana Network Troubleshooting 74

gement
 How monitor one machine?
• You are asked to check out a server on a switched network: what to
do?
• Use a small hub, and use a notebook running the capture software
Ethernet
Switch

mini-hub

Device under test notebook

e.g., a server running capture software

Systems and Network Mana Network Troubleshooting 75

gement
 Are switched networks secure?
• Is all unicast traffic on one port of a
switch private?
• No, there are tools (dsniff) freely
available to temporarily make a switch
behave like a hub, or that provide
other ways to compromise switch
security.

Systems and Network Mana Network Troubleshooting 76

gement
 Port Scanning

Identify services offered by a

remote computer
 What is a port scanner?
• Sends packets to various ports on a
network device
• Best one available everywhere is nmap
• can identify the OS of the target machine
• Do not port scan arbitrary machines in your
company's network without permission!
• May be interpreted as a cracking attempt

Systems and Network Mana Network Troubleshooting 78

gement
 How does nmap identify OS?
• RFCs leave interpretation of some things up to the
implementer
• RFCs do not specify how should work if get
contradictory flags, strange sequences of
inconsistent packets
• Most TCP/IP implementations are not complete
• Every implementation of TCP/IP is different; the
“grey areas” are different from one OS to another.
• nmap sends “strange” packets to the machine,
detects how reacts, matches this against a file of
OS fingerprints

Systems and Network Mana Network Troubleshooting 79

gement
 Running nmap: Use xnmap
• $ sudo –v
• $ sudo xnmap &
• Enter the IP address of machine(s) to
identify
• select other choices from buttons
• press Start
• xnmap is simply a way to easily generate
command line options to nmap using a
graphical interface

Systems and Network Mana Network Troubleshooting 80

gement
 Uses of nmap
• Identify the type of a computer that
is causing trouble on the network
• Check what network services a
computer is really offering
– compare with netstat output
– A cracked computer may be hiding some
services with trojaned utilities
– nmap can help you discover such services
Systems and Network Mana Network Troubleshooting 81
gement
Troubleshooting Protocols

DNS
email
using telnet
 DNS troubleshooting
• Suspect DNS when get long timeouts
before see any response
• ping name, IP address, see if only IP
address works
• tools on Linux, Unix:
– dig, nslookup, host
• tools on Windows:
– nslookup

Systems and Network Mana Network Troubleshooting 83

gement
 DNS: dig
• The people who write the most
common name server (Bind) promote
dig, deprecate nslookup
• dig output is in form of DNS
resource records
– can copy and paste straight into DNS
database files

Systems and Network Mana Network Troubleshooting 84

gement
 dig: Checking forward DNS lookup
$ dig sysadmin.no-ip.com

; <<>> DiG 9.2.1 <<>> sysadmin.no-ip.com

;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23568
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3,
ADDITIONAL: 3

;; QUESTION SECTION:
;sysadmin.no-ip.com. IN A

;; ANSWER SECTION:
sysadmin.no-ip.com. 60 IN A 202.69.77.139

Systems and Network Mana Network Troubleshooting 85

gement
 dig: checking forward DNS 2
;; AUTHORITY SECTION:
no-ip.com. 60 IN NS nf1.no-ip.com.
no-ip.com. 60 IN NS nf2.no-ip.com.
no-ip.com. 60 IN NS nf3.no-ip.com.

;; ADDITIONAL SECTION:
nf1.no-ip.com. 60 IN A 66.185.166.131
nf2.no-ip.com. 60 IN A 66.185.162.100
nf3.no-ip.com. 60 IN A 216.66.37.10

;; Query time: 254 msec

;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Feb 24 10:55:26 2003
;; MSG SIZE rcvd: 154

Systems and Network Mana Network Troubleshooting 86

gement
 dig: reverse lookup 1
$ dig -x 202.69.77.139

; <<>> DiG 9.2.1 <<>> -x 202.69.77.139

;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22117
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;139.77.69.202.in-addr.arpa. IN PTR

;; ANSWER SECTION:
139.77.69.202.in-addr.arpa. 3600 IN PTR 077-139.onebb.com.

Systems and Network Mana Network Troubleshooting 87

gement
 dig: reverse lookup 2
;; AUTHORITY SECTION:
77.69.202.in-addr.arpa. 3600 IN NS ns2.onebb.com.
77.69.202.in-addr.arpa. 3600 IN NS ns1.onebb.com.

;; Query time: 310 msec

;; SERVER: 172.19.64.52#53(172.19.64.52)
;; WHEN: Mon Feb 24 11:07:04 2003
;; MSG SIZE rcvd: 111

Systems and Network Mana Network Troubleshooting 88

gement
 dig syntax
dig [options] [@server] name type
• main option is –x
• server is the name server to query
– by default, use first server in
/etc/resolv.conf
• name is what you want to look up
• type can be: any, a, mx, axfr, soa, etc
• default is to get A record(s)
Systems and Network Mana Network Troubleshooting 89
gement
 dig: axfr (Zone Transfer)
• dig can request a complete zone
transfer:
dig @ictlab tyict.vtc.edu.hk axfr
• result can be copied and pasted as a
master file in a DNS server

Systems and Network Mana Network Troubleshooting 90

gement
 nslookup: an interactive program
$ nslookup
Note: nslookup is deprecated and may be removed
from future releases. Consider using the `dig'
or `host' programs instead. Run nslookup with
the `-sil[ent]' option to prevent this message
from appearing.
> sysadmin.no-ip.com
Server: 127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
Name: sysadmin.no-ip.com
Address: 202.69.77.139

Systems and Network Mana Network Troubleshooting 91

gement
 nslookup: reverse lookups
> 202.69.77.139
Server: 127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
139.77.69.202.in-addr.arpa name = 077-
139.onebb.com.

Authoritative answers can be found from:

77.69.202.in-addr.arpa nameserver = ns1.onebb.com.
77.69.202.in-addr.arpa nameserver = ns2.onebb.com.
ns1.onebb.com internet address = 202.180.160.1
ns2.onebb.com internet address = 202.180.161.1
>

Systems and Network Mana Network Troubleshooting 92

gement
 Email: testing with telnet
• Email protocols SMTP, POP3 are text
• telnet a good tool to test them
• syntax:
• telnet server portnumber
• SMTP: port 25
• POP3: port 110

Systems and Network Mana Network Troubleshooting 93

gement
 Test the VTC mail server:
$ telnet smtp.vtc.edu.hk 25
Trying 192.168.79.191...
Connected to smtp.vtc.edu.hk (192.168.79.191).
Escape character is '^]'.
220 pandora.vtc.edu.hk ESMTP Mirapoint 3.2.2-GA; Tue, 25 Feb 2003
11:15:30 +0800 (HKT)
helo nickpc.tyict.vtc.edu.hk
250 pandora.vtc.edu.hk Hello [172.19.32.30], pleased to meet you
mail from:<[email protected]>
250 <[email protected]>... Sender ok
rcpt to:<[email protected]>
250 <[email protected]>... Recipient ok
data
354 Enter mail, end with "." on a line by itself
My message body.
.
250 AFF21826 Message accepted for delivery
quit
221 pandora.vtc.edu.hk closing connection
Connection closed by foreign host.

Systems and Network Mana Network Troubleshooting 94

gement
 SMTP commands for sending mail
• helo — identify your computer
• mail from — specify sender
• rcpt to — specify receiver
• data — indicates start of message
body
• quit — terminate session
• Use names, not IP addresses, to
specify destination
Systems and Network Mana Network Troubleshooting 95
gement
 Testing the VTC pop3 server 1
$ telnet pop.vtc.edu.hk 110
Trying 192.168.79.12...
Connected to pop.vtc.edu.hk (192.168.79.12).
Escape character is '^]'.
+OK carme.vtc.edu.hk POP3 service (iPlanet Messaging Server
5.2 Patch 1 (built Aug 19 2002))
user nicku
+OK Name is a valid mailbox
pass password
+OK Maildrop ready
stat
+OK 1 673

Systems and Network Mana Network Troubleshooting 96

gement
 Testing the pop3 server 2
retr 1
+OK 673 octets
Return-path: <[email protected]>
Received: from pandora.vtc.edu.hk (pandora.vtc.edu.hk [192.168.79.191])
by carme.vtc.edu.hk (iPlanet Messaging Server 5.2 Patch 1 (built Aug 19 2002))
with ESMTP id <[email protected]> for nicku@ims-ms-daemon
(ORCPT [email protected]); Tue, 25 Feb 2003 11:16:29 +0800 (CST)
Received: from nickpc.tyict.vtc.edu.hk ([172.19.32.30])
by pandora.vtc.edu.hk (Mirapoint Messaging Server MOS 3.2.2-GA)
with SMTP id AFF21826; Tue, 25 Feb 2003 11:16:01 +0800 (HKT)
Date: Tue, 25 Feb 2003 11:15:30 +0800 (HKT)
From: Nick Urbanik <[email protected]>
Message-id: <[email protected]>

My message body.
.
dele 1
+OK message deleted
quit
+OK
Connection closed by foreign host.

Systems and Network Mana Network Troubleshooting 97

gement
 pop3 commands: retrieving mail
• See RFC 1939 for easy-to-read details
• First, must authenticate:
• user username
• pass password
• stat — shows number of messages and total size in bytes
• list — list all the message numbers and size in bytes of
each message
• retr messagenum — retrieve the message with number
messagenum
• dele messagenum — delete the message with message
number messagenum
• quit

Systems and Network Mana Network Troubleshooting 98

gement
 Telnet: Testing Other Applications
• Many network protocols are text. Telnet can
be helpful in checking:
• IMAP servers:
– telnet hostname 143
• Web servers:
– telnet hostname 80
• Ftp servers:
– telnet hostname 21
• Even ssh (can check version, if responding):
– telnet hostname 22
Systems and Network Mana Network Troubleshooting 99
gement
 Conclusion
• Check the simple things first
• Document what you do
• Become familiar with common tools
• Use the tools to become familiar with your
network before troubles strike
• Know what is “normal”
• Get permission from the boss before using
packet sniffing and port scanners

Systems and Network Mana Network Troubleshooting 100

gement