The 21-Second Mystery

One of the most intriguing revelations of the examination of this SubSeven traffic was the 21-second time preceding the peak activity for the initial two scans, and later a third, that were observed. It was clear that there was some meaning and explanation associated with this; this couldn't be a mere coincidence because it occurred three times.

I have an annoying habit: When I'm stumped and frustrated by my inability to figure something out, I start plaguing colleagues. Most have learned to dismiss me with some plausible excuse like, "There are free donuts in the cafeteria. See you later." But, I cornered my co-worker and longtime bicycling buddy, Vern, and asked him to ponder this mystery. Within seconds, and still a good chance to get those cafeteria donuts, he said, "Oh, that's easy; it's the combined backoff times for retries." This insight made us rethink our approach, and we eventually plotted the traffic separately for initial SYNs and retries, allowing us to discover that the 21-second peak rate was an overlap of retries from different initial waves of SYN activity.

Fingerprinting Participant Hosts

The assumption now is that the zombie hosts have been "infected" with some malware that is generating the scanning activity. The question then becomes this: Is there a specific operating system that has been exploited, transforming the host into a zombie for this scan? An examination of passive fingerprints can assist in identification of zombies' operating systems. This assumes that the packets coming from these hosts are not crafted to change default values, such as TCP window size, initial TTL, and TCP options.

Passive fingerprinting categorizes operating systems by looking at unique field values in the packets that have been sent. As we have discussed, different operating system TCP/IP stacks choose unique values for certain fields, such as Time to Live (TTL), TCP window size, and TCP options. There are also other fields that can be examined, such as the Type of Service (TOS) value and the don't fragment (DF) flag. But, because most operating systems use a default TOS value of 0 and set the DF flag, this might only determine the small percentage of unusual values sent from other operating systems. And, these two fields are best examined in conjunction with other fields and not alone.

Table 11.1, provided by the Honeynet Project, was used in determining some of the scanning hosts' operating systems. The lines that are highlighted represent the operating system and associated fingerprints of the majority of the scanning hosts that were observed for this activity.

	Table 11.1. Passive Fingerprinting Values by Operating System
# OS	VERSION	PLATFORM	TTL	WINDOW	DF	TOS
#---	-------	--------	---	-----------	--	---
DC-OSx	1.1-95	Pyramid/NILE	30	8192	n	0
Windows	9x/NT	Intel	32	5000-9000	y	0
NetApp	OnTap	5.1.2-5.2.2	54	8760	y	0
HPJetDirect	HP_Printer		59	2100-2150	n	0
AIX	4.3.x	IBM/RS6000	60	16000-16100	y	0
Cisco	11.2	7507	60	65535	y	0
DigitalUnix	4.0	Alpha	60	33580	y	16
IRIX	6.x	SGI	60	61320	y	16
OS390	2.6	IBM/S390	60	32756	n	0

Figure 11.4 for June 29, 2001 shows that there are three clusters of arriving TTL

Reliant	5.43	Pyramid/RM1000	60	65534	n	0
FreeBSD	3.x	Intel	64	17520	y	16
JetDirect	G.07.x	J3113A	64	5804-5840	n	0
Linux	2.2.x	Intel	64	32120	y	0
OpenBSD	2.x	Intel	64	17520	n	16
OS/400	R4.4	AS/400	64	8192	y	0
SCO	R5	Compaq	64	24820	n	0
Solaris	8	Intel/Sparc	64	24820	y	0
FTX(UNIX)	3.3	STRATUS	64	32768	n	0
Unisys	x	Mainframe	64	32768	n	0
NetWare	4.11	Intel	128	32000-32768	y	0
Windows	9x/NT	Intel	128	5000-9000	y	0
Windows	2000	Intel	128	17000-18000	y	0
Cisco	12.0	2514	255	3800-5000	n	192
Solaris	2.x	Intel/Sparc	255	8760	y	0

This table of information was obtained at http://project.honeynet.org/papers/finger/traces.txt.

Arriving TTL Values

If you recall, the arriving TTL values can be used to help identify the scanning host's operating system. Different operating systems use different initial TTL values when sending a packet. Each router through which the packet travels on its journey from source to destination host examines the TTL value and decrements it by 1. This becomes an indication of the number of "hops" that the packet has traveled. If a router ever discovers a TTL of 0, it discards the packet and sends back an ICMP error message of "time exceeded in-transit" to the sending host. This informs the sending host that the packet has exceeded its welcome on the Internet. This is a mechanism that is used to discard lost packets, such as ones that have become caught in a routing loop.

Initial TTLs of many operating systems have typical values of 32, 64, 128, and 255. These might be different per protocol—TCP, UDP, or ICMP. For instance, Windows NT 4.0 Service Pack 6 has an initial TTL value of 128 for TCP and an initial TTL value of 32 for ICMP packets sent. Fortunately, this examination is limited to TCP so there is no need to account for protocol differences. The arriving TTL values are examined and are helpful in estimating the initial TTL values. The caveat here is that although most operating systems will be configured to use the default initial TTL values, these can be changed. All that can be determined with absolute certainty from the arriving TTL is that it is less than the initial TTL. Of course, this assumes that the source host and destination host are not directly connected to the same local network, in which case the packet could pass from source to destination without the TTL being decremented.

Examination of

values for the scans. More specifically, the closest scanning host appears to be 8 hops away, and the most distant appears to be 25 hops away from the capturing sensor interface. The assumption is that the scanning hosts had initial TTL values of 128, 64, and 32, and the arriving TTL values are associated with an initial TTL value that is greater than the initial TTL value by the least amount. For instance, if an arriving TTL is 50, it is assumed to have an initial TTL value of 64 and not 128, although either initial TTL value would be valid.

Figure 11.4. June 29, 2001 arriving TTL values.

Figure 11.5 for July 2, 2001 shows the same clustering. More specifically, the

In the June 29 scan, the largest percentage of scanning hosts, 92.13, had an estimated initial TTL of 128. More than 37 percent of the hosts with an initial TTL of 128 were approximately 11 to 13 hops away from the sensor. According to Table 11.1, an initial TTL value of 128 is indicative of Windows 9x/NT/2000. An initial TTL value of 32 is Windows 9.x/NT, which comprised 2.66 percent of the scanning hosts. The initial TTL value of 64 is associated with many of the UNIX platforms, including the Linux 2.2.x kernel. The percentage of hosts with an initial TTL of 64 was 5.2.

Examination of

closest scanning host appeared to be 8 hops away, and the most distant appeared to be 27 hops away from the capturing sensor interface.

Figure 11.5. July 2, 2001 arriving TTL values.

Looking at the July 2 scan, the largest percentage of scanning hosts, 92.29, had an initial TTL of 128. More than 37 percent of the hosts with an initial TTL of 128 were approximately 11 to 13 hops away from the sensor. 2.36 percent of the scanning hosts had an initial TTL of 32. Finally, 5.35 percent of the scanning hosts had an initial TTL of 64.

The determination from this is that the scanning hosts are not exclusively Windows hosts, but it appears that Windows hosts are the majority of the scanners. This means that whatever malware is exploiting the scanning hosts, it is not exclusive to Windows.

Although the x-axis scaling for plots in Figures 11.4 and 11.5 doesn't readily show this, there was a very distinct clustering around the estimated initial TTL values. For instance, in the June 29 scan, there is a noticeable gap or absence of packets with arriving TTL values between 22 and

42 and between 56 and 103. Similar behavior is observed for the July 2 scan.

TCP Window Size

A host advertises the TCP window size when it attempts to make an initial connection. The window size is a dynamic value that changes as information is exchanged between hosts and represents the current TCP buffer size for the incoming data. This buffer allows multiple packets to be sent and queued before passing them to TCP and the application. More simply, a given

operating system has a default value for the TCP window size, and the window size can change dynamically as data is received and processed.

But, the initial window size can be used to fingerprint the operating system. The user or administrator can customize this, but commonly the default is used.

As you can see in Figure 11.6, the bulk of the connections had an initial window size of 8192. This is associated with Windows 9x/NT connections according to Table 11.1. Although the table doesn't have a window-size entry for 16384, research has discovered it is associated with Windows 2000. Table 11.1 alludes that a window size of 65535 is associated with Cisco. However, it appears that the high percentages associated with this window size would include other operating systems.

Figure 11.6. Scanning host TCP window size.

Search engines on the Internet failed to find any operating system associations with a window size of 65535. Attempts were made to examine a week's collection of TCPdump data for the monitored site to find hosts that had a window size of 65535. Only a dozen of approximately 5,500 hosts were found with a window size of 65535. A scan by nmap could not determine the operating systems. Some of the hosts had ports open, such as 135 and 139, which would indicate Windows versions prior to Windows 2000. Others had port 445 listening, which was introduced in Windows 2000 to support Server Message Block (SMB) talking directly over TCP/IP without the need for the intermediate layer of NetBIOS over TCP/IP (NBT). Yet, other hosts with a window size of 65535 listened at ports 111 (portmapper), 515 (line printer daemon), and 6000 (X11), which are all associated with UNIX hosts. No conclusions could be reached about the operating system associated with a window size of 65,535 based on these findings.

Other unique window sizes that were seen were 32120, associated with Linux, which was found in the June 29 scan only and comprised .19 percent of the total scanning hosts. A window size of 8760 was seen in both scans and reflects a Solaris host. The first scan had 5.21 percent