Network tools

From Helpful
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

For other network related things, see:


Also:

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Medium level tools

Download tools

You probably already know wget or curl. In that context, lftp is also interesting.

  • wget is a HTTP and FTP downloader, with some neat features geared towards those protocols
  • curl is like wget, with a different feature set and supporting some more protocols
  • lftp is a useful (mass) downloading tool that does HTTP, FTP, SFTP, and others

See also this comparison table


Low-level tools

Pings and port scans

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

nmap notes

(assumption is use on home LAN. For security stuff, learn it properly)


Nmap port states

  • open:
computer is listen()ing and will accept() connections on port
  • closed:
host responds with ICMP 'Port Unreachable' message
probably because it's not listen()ing on port
those ICMP packets can also be generated by firewall rules - meaning that this port may be closed to you but open to others
  • filtered can mean a few things:
got some ICMP unreachable errors that aren't Port Unreachable (net unreachable, host unreachable, etc.)(verify)
OR when there is no response after some time, e.g if the packets were dropped (at the host, or a router on the way)
  • unfiltered:
most other ports were filtered, but this one is closed (verify)


Note that different ICMP rejections in iptables can lead to either 'closed' or 'filtered', so ports may still be open to other places (or under other conditions).

Some arguments

  • -Tn: aggressiveness/speed. See [1]
0 and 1 will avoid most intrusion detection, but also take very long
Default is -T3
-T4 and -T5 are often acceptable on LAN
-T4 may affect your internet access, -T5 is a bit asocial and may affect accuracy. Both are likely to be blocked if there is any intrusion detection


Quick (and approximate) host discovery:

  • -sP, later -sn
    • Ping only, no port scan after it
    • won't report hosts not responding to ping
nmap -sP 192.168.1.0/24


Interesting arguments:

  • -p: port(s) to scan on.
By default it scans 1000 commonly used ports
with -F it scans 100 commonly used ports
If you're interested in a faster scan and only a single service, e.g. -p 22
...or finding hosts with a bunch of services, e.g. -p 1-200
...or a few specific things, say -p 80,22,5353,445,139,631,123,3689,548,7,5900,5000
...or really everything, with -p 0-65535
Note that scans on the same subnet are a lot faster than those elsewhere


Target spec

I commonly use 192.168.2.0/24 style

Other options:

  • example.org/24 - mask is applied after resolving the name to an IP
  • list and ranges like 1,20-200,250-254, probably but not necessarily in the last octet
192.168.2.20-200
192.168.0-255.1
192.168.0,1,2,100.1,20-200,254


There are more tricks, see http://nmap.org/book/man-target-specification.html


"Host is down. If it is really up, but blocking our ping probes, try -Pn"

Watching traffic

Speed per established connection:

  • jnettop (connections, speeds)
  • iftop (connections, speeds) - graphical feedback of speeds
  • tcptrack (connections, speeds)
  • iptraf (per connection / interface / protocol / MAC)


Speed per program:

  • nethogs (speeds per program; by default only TCP)


Speed totals (per interface):

  • bmon (speed per interface, and shows traffic shaping aggregates)
  • nload (graphs)
  • vnStat (also summarizes per day, etc.; collects via background service)
  • iptraf (per connection / interface / protocol / MAC)
  • ibmonitor (speed per interface)
  • (slurm(verify))


Connections:

  • ssldump lets you notice SSL traffic (and decrypt it, given the right keys)


Packet-level

  • tcpdump takes packets from the network stack, and (with default options) gives you a short description. Also allows you to filter, write packets to the tcpdump file format (various utilities can read this). See also tcpdump notes.
  • Wireshark (previousy ethereal)[2], which still exists but isn't being developed anymore) is similar but has a GUI, some more filter options (a different filter system), and more advanced packet decoding.
  • tcpflow: instead of storing packets, this stores TCP connection interchanges in whole, each each in a separate file (or optionally only to screen). Doesn't write a standard file format, but is useful to snoop on protocols at application layer
(can be useful to split interchanges from a tcpdump file)


Content-geared:

  • ngrep greps packets for contents. Can e.g. be used as a content-aware tcpdump alternative, in a pipe, or to filter tcpdump files after the fact.
  • driftnet picks out images from HTTP transfers and either saves them or displays them in X.
  • chaosreader interprets packet log files and snoops out files, emails, etc. from the common protocols that carry them (HTTP, FTP, SMTP)
  • dsniff is like chaosreader, but a little lower-level.



arpwatch

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Roughly speaking, this reports when observed ARP packets give new or interesting information

It stores MAC and IPs that we have seen (up to half a year?), to support certain detections

Indirectly helps detect address reuse, ARP spoofing.

It can be told to mail certain reports.

(By default listens on a single interface, the first one it finds. It seems arpwatch services may be configured to launch one per interface?)


It reports

  • "new station" when it sees a MAC we hadn't seen (...in the last half year)
  • "new activity" when it sees a (MAC,IP) pair that we know, but hadn't seen for half a year(verify)
so also when you change or add an IP to an interface(verify)
  • "changed ethernet address" -
  • "flip flop" - address changed from the most recently seen address to the one before that (of rememembered addresses)
  • "reused old ethernet address" - address changed to the third-or-more (of rememembered addresses)
  • bogons for any IP address that isn't in our the monitored interface's (first?) subnet
you can disable this with -N
you can add networks you expect traffic from with -n


You can ignore IP ranges with -z

tcpdump notes

This article is marked 'feel free.' Often because its authors know they won't spend as much time on it as they should. Your help is appreciated.

TCPdump snoops network traffic going getting to the host it runs on.

It can also filter what packets it shows/writes.

It can summarize the traffic on-screen (the default) or write it to the fairly common pcap-style files.

The 'tcp' in the name is misleading, because it suggests it captures at the network layer, and then specifically TCP. Actually, it captures at link layer, so you can get at a lot of interesting stuff.


Wireshark (previously Ethereal) serves much the same purpose. It has a GUI, more protocol decoding, a more advanced filter language that includes substring and regexp content filtering.

You may also be interested in tshark, which can do more than tcpdump


Usage notes

If the line reporting "packets dropped by kernel" doesn't say 0: This refers to to the packet capture interface (a kernel feature) and doesn't mean the kernel is screwing up your networking, nor does it refer to firewall drops.

The network is moving packets faster than the capturing program is reading them from the capture interface. That is, the kernel isn't waiting around for the capturer (tcpdump) to catch up, it's removing the packets exposed to the capturing.

It seems one reason is that tcpdump is waiting on its own DNS lookups - try tcpdump -n (numeric, no names) to see if that helps.


Interesting command line options

  • -n: don't resolve hostnames (faster)
  • -i interfacename: listen on a specific interface. Default is the first it can find. ("-i any" will capture from all interfaces but ''not'' be promiscuous)


  • -w filename: dump packet data to a file, instead of decoding and printing (tcpdump -r and other utilities take this)
  • -s 0: capture the whole packet instead of the first 68 bytes (68 is faster and enough for the scrolling screen summary, but not for deeper inspection. You probably want whole packets when writing traffic to files for later inspection)


  • -t: don't show timestamps (less clutter, though also not as informative)

Print packet as

  • -A: ASCII (safe to print; command characters are removed)
  • -x: hex
  • -X: hex and ASCII
  • (-xx and -XX are variations that also show the link-level header)

Filter expressions

The filter expression is often enclosed in single quotes. This is not necessary, but is simpler than dealing with shell escaping.


Note that the predefined primitives require a backslash before them.

Apparently the format is the one from BPF, so see things like [3]


Host, net, port

Probably the most directly interesting types filters are things like:

  • port 53 (matches tcp and udp; "udp port 53" and "tcp port 53" are shorthands that imply the respective protos)
  • portrange 1024-65535
  • host 192.168.1.1 (shorthand for "ether proto \ip and host ...")
  • net 192.168 (shorthand for "ip net ..."), and also:
    • net 192.168.5 mask 255.255.255.0
    • net 192.168.5/24

When not mentioning src or dst, you accept an address, net, port or such if it appears as either source or destination. So: port 80 would capture your surfing and people connecting to your web server, while src port 80 and dst port 80 look at direction.


If you add a named hosts it is looked up, but redirects and round robin DNS resolving may throw you off (both may happen in the case of google.com).

Protocol constants
  • "ether proto" things
    • ip (shorthand for "ether proto \ip", and actual value is 0x0800)
    • arp (shorthand for "ether proto \arp", value is 0x0806)
    • rarp ("ether proto \rarp")
    • ip6, tr, fddi, decnet
    • Others, e.g. 0x8863 for PPPoE Discovery, 0x8864 for PPPoE Session

Example:

  • ICMP other than pinging: tcpdump 'icmp[icmptype]!=icmp-echo and icmp[icmptype]!=icmp-echoreply'
  • "ip proto" things:
    • tcp (shorthand for "ip proto \tcp")
    • udp (shorthand for "ip proto \udp")
    • icmp (shorthand for "ip proto \icmp")
    • Also: gre, ipx, decnet

(note the absence of backslashes in shorthands)

Logic in filters

Logical booleans:

  • and, &&
  • or, ||
  • not, !

Examples:

  • 'not arp and not port 67' (show all except arp and dhcp)
  • 'net 192.168 && !host 192.168.1.1' (all traffic on this net not related to this specific host)
  • 'net 192.168 || net 145.99' (traffic on my private and public IP net)


Comparison:

  • eq, == and =
  • ne, !=
  • gt, >
  • lt, <
  • ge, >=
  • le, <=
Repetition and omission: lists and short forms

If the predicate is missing, the last is assumed. For example:

net 10 or 192

...does the same as

net 10 or net 192

Lists are supported through a similar sort of expansion:

host ourrouter and not \( ourmailserver or ourwebserver \)


Raw bit/byte testing, and bitwise operations

You can get at packet data, with decimal (or hexadecimal) offsets for a frame/packet at apparently any layer/decoded protocol, including things like ether[], link[], ip[], tcp[], udp[], and icmp[]. Some examples:

  • ip[8] = 1 (picks out a byte. This tests for TTL being 1)
  • tcp[0:2] = 80 (picks out a word using [start offset:length]. This one is equivalent to "src port 80")
  • icmp[0]==8 || icmp[0]==0, ICMP packets that are either echo request or echo reply. Note that there are readable constants for these, used in an example somewhere above.
  • tcp[20:4] = 0x48454C4F (one page's example; those are ASCII bytes for HELO, part of SMTP mail sending)
  • port 80 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420 HTTP GET - but note that such trick get awkward fast and may not work well when there is any variation you didn't consider

See links below for header references, and for further examples using this sort of test.


Value/bitwise operations that may be useful:

  • +, -, /, and *: integer math
  • & and |: bitwise and and or
  • <<, >>: bitshifts

Note that you only ever deal with whatever endianness is defined for the protocol. IP, TCP and UDP are always big-endian.

TCP flags
tcp-urg
tcp-ack
tcp-push
tcp-rst
tcp-syn
tcp-fin

e.g. to show any packets with any of these three flags:

'tcp[tcpflags] & (tcp-syn|tcp-rst|tcp-fin) != 0'
Length

Also interesting in this context is len, the packet packet length. I'm not sure at which level -- it seems that on Ethernet, filtering for 1514 gives results and not 1500 or 1518, which would suggest it's counting at link level, except it's NOT counting the CRC.

Copy-paste examples

  • ICMP packets: tcpdump icmp (ping and ICMP portscans)
  • non-ping ICMP: tcpdump 'icmp[0]!=8 && icmp[0]!=0' (often mostly 'port unreachable' replies)
  • ARP packets: tcpdump arp
  • ARP and DHCP: tcpdump 'udp port 67 or udp port 68 or arp'
  • SYN packets: tcpdump '(tcp[13]&0x2)!=0 and (tcp[13]&0x10)==0' (initial establishment of TCP connections, SYN portscans) (Specifically packets with SYN set, ACK not set, and ignoring the other flags)


  • MAC filter: tcpdump 'ether host 00:0B:6A:11:22:33' (also see ether src and ether dst)
  • Ethernet broadcasts: tcpdump 'ether broadcast' (shows ARP requests and more)
  • Small data packets: tcpdump '(tcp or udp) and len<100'
  • IPv6: tcpdump ip6



There are various tricks you can pull when you combine other networking tools, or even general tools. For example, you can add pv to see how much data of a specific type (say HTTP) is being moved (at ethernet level):

tcpdump 'tcp port 80' -s 0 -w - | pv > /dev/null

Wifi

Various wireless-related utilities dump or read the pcap format, so you can use tcpdump to read the results.

BSSIDs act as MACs, so you can filter for traffic to and from an AP using:

ether host 00:12:34:66:22:12

Tcpdump knows how to decode the 802.11 (WiFi) basics, but there seem to be no handy aliases yet. Since 801.11 is a different type of ethernet frame (the aliases seem for ethernetII), most of the interesting filters have to come from accessing the packet's link level data. See also wireless.

Interesting tests include

  • link[0]==0x80: Beacon
  • link[0]&0x0c==0x08: Data packets
    • link[0:2]&0x0c02==0x0802: WEP-encrypted data packets

See also

  • tcpslice, which manipulates dump files


Examples:

Header reference / byte/bit-tests

Reference:


wireshark and tshark notes

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Wireshark captures network data and shows it visually. It's e.g. a great network debugging tool.


tshark is basically the command line variant of wireshark. This makes it like tcpdump, but a little more capable.


wireshark and tshark have a slightly confusing difference between capture filters and display filters

Capture filters

are what can be passed into libpcap/winpcap,
so is the same syntax that tcpdump, WinDump, and others use
cannot be changed during a capture
are intended mostly to limit the amount of data we pick up into RAM / onto disk (which is great on busy networks)


Display filters

are much more capable
if the program allow s(e.g. wireshark GUI), you can alter while capturing
only change what part of already-captured data is being shown right now (so can be changed live)


Yes, there is plenty of overlap - with different syntax, e.g.

tcp port 80 (capture filter) versus
tcp.port == 80 (display filter)


In wireshark you get asked at different times.

In tshark it's mostly:

  • -f <capture filter>
  • -Y <displaY filter>

...there are actually a few more filter related options, most of which are only relevant for more advanced use.


You'll probably frequently use capture filters to narrow to what you're interested in, which at first may be as broad as things like:

net 192.168.0.0/24
src net 192.168.0.0/24
dst net 192.168.0.0/24
host 172.18.5.4

ether broadcast or ether multicast
multicast and not broadcast

net 192.168.0.0/24 or net 0.0.0.0/8 or 224.0.0.0/4

arp

port 67 or port 68


See also:


Display filters

Display filters try to expose a lot of useful things as fields.


See also:


Complex tricks

Reports and statistics

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

System/traffic reports

  • darkstat (standalone: snoops off interface, reports via embedded web server and simple graphs)
  • ntop does traffic analyses and rrd-style graphs (host-focused)

See also this list

Availability/health monitoring


Log analysis

(See also Web log analysis notes)


Lower level tools

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

(relatively lower, or very-specific-purpose)

Utilities

  • lft ('layer four traceroute') [5]
  • bing
  • dnstracer

netcat

There are two distinct things called netcat

  • the original [9]
  • the BSD implementation [10]

The idea is the same but some behaviour differs, e.g. when things are output.

(Note that nc is sometimes symlink to mc, the midnight commander, because it's an imitation of norton commander)


Uses of netcat include talking to your web server:

# the -e is there so that \n gets interpreted as a newline
echo -e 'GET / HTTP/1.0\n' | netcat localhost 80


The utility can also create a listening (-l) TCP server (default) or UDP server (-u option) on a port, say 1111 (-p 1111):

netcat -l -p 1111
netcat -l -u -p 1111

These will echo what they get. Since this uses stdin/stdout, this can be used for simple network pipes.

netcat servers can be useful to see whether connections are getting through, and see what's happening on them (try a lot of verbosity through -vv), which can be useful e.g. in protocol testing and testing whether your firewall is being overly protective.


You can do some basic port scanning, for example with a one-second timeout after connect, port 10-500)

netcat -v -w 1 localhost -z 10-500


things like netcat

Variations on netcat include:

generalized in that it can connect to a file, pipe, device, socket (Unix, IP4, IP6, raw, UDP, TCP), SSL, and some proxies
and can relate between these
http://freshmeat.sourceforge.net/projects/socat/
  • nmap's ncat
https://nmap.org/ncat/
https://nmap.org/ncat/guide/index.html


  • pnetcat - python implementation of netcat
http://stromberg.dnsalias.org/~strombrg/pnetcat.html


  • the /dev/tcp device, see e.g. [12]

More specialized

  • cryptcat
adds (twofish) encryption, so that you can transfer data without it being easily snoopable(verify)
http://cryptcat.sourceforge.net/
  • netcopy & netsend - plain file sending


  • packet sender
https://packetsender.com/
  • hping
https://en.wikipedia.org/wiki/Hping


Dead homepage?

  • sbd
sometimes noted to be a trojan, mostly because it allows remote execution
http://www.cycom.se/dl/sbd
  • emcast (sort of a multicast netcat)



Packet creation

Speed limit/stress testing



IP-to-location lookup

hostip

http://www.hostip.info/

Looks interesting and detailed, though it seems that you'll have to your own indexing.

Maxmind GeoIP

Has decent-resolution free (requires attribution) data sets, GeoLite:

There are better-resolution versions for pay.

Caching options:

  • None: GEOIP_STANDARD: reads everything from disk
  • Index only: GEOIP_INDEX_CACHE: index stored in memory, record read form disk: faster than standard, less memory usage than full. Often the smart choice for the more detailed datasets (like city)
  • Full: GEOIP_MEMORY_CACHE: Everything is loaded once and ket in memory and GEOIP_CHECK_CACHE same, but check the filesystem whether the db has changed and reload if so

IP2Location

http://www.ip2location.com/

Seems annoyingly paid-for.