3 Intrusion detection and prevention systems (ids/ips)

Intrusion detection/prevention is the process of monitoring and defining unauthorized access attempts. IDS/IPS systems are advanced mechanisms of "in-depth" protection allowing for the detection and prevention of potentially dangerous "events" [15]. Some examples of these events" are:

- Buffer overflow;

- Executing of a malicious code (worms, viruses);

- Worm propagation;

- Illegal data control;

- Exploitation of a programs vulnerabilities.

IDS/IPS can be implemented in two approaches host and network-based IDS/IPS. Additionally, these approaches have two realizations which are signature and anomaly-based IDS/IPS which will be considered below.

3.1 Signature-based ids/ips

Signature-based IDS/IPS is the most accurate technique to detect well-known attacks. Representations of intrusions and attacks are stored as signatures and then are compared to the system activity in order to detect and prevent "abnormal" activity. Signature-based analysis is similar to the analysis that an antivirus program may provide which allows the detection of malicious content in the packet payload.

Signatures are a specific set of rules or a sequence of bytes derived from each particular attack's pattern. A signature can contain a malicious code pattern, commands to open shell command line or a malicious combination that is used for buffer overflow attacks [15].

This type of IPS/IDS has the following benefits:

- provide the full information about an attack as a result can stop it correctly;

- detect malicious content, shortcut combinations in the network packets;

- stop and prevent dDoS and inside/outside attacks;

- detect access to files and objects;

- provide a connection control;

- control e-mail, IM content;

- Less false positives;

- Specific checks can be implemented (in accordance with signatures);

Despite the advantages, the signature-based approach has the following set of disadvantages:

- Can not to detect zero-day and stealth attacks;

- Signatures must be created for each attack (impossible to use common features);

- can be blocked by network traffic or in different segments of network (if it is distributed);

- Vulnerable to "false" generation attacks;

- Initially, a lot of false positives/negatives;

- needs a lot of memory resources;

- Each attack pattern must be fully determined;

- 1 byte difference and an attack can not to be detected;

3.2 Anomaly-based ids/ips

The main point of anomy-based technique is to define baseline parameters for network behavior. During the operation, system defines the network behavior and any deviation from the baseline parameters if detected triggers and alarm. The main purpose of such analysis is to detect or provide an opportunity to detect the wide range of attacks that have no-predefined attributes (signatures) [15]. Anomaly-based IDS have no dependence on knowledge about vulnerabilities in a particular network environment which allows them to detect and prevent unknown attacks (e.g.- zero-days and stealth attacks). Anomaly detection is a static process which is based on the concept of determining the number of events that occur in a certain time period (based on size and traffic load of the network) for a particular behavior pattern of intrusion.

The anomaly-based approach has the following advantages allowing for the detection and prevention of:

- Zero-days attacks;

- Abnormal user activity (internal anomaly detection);

- Escalation and unauthorized access attacks;

- Intrusion in any type of networks (has no dependence on network size).

However, this approach is not without drawbacks such as:

- creates a lot of "false" positives;

- Do not provide the fully described information about an attack (only deviations from predefined parameters);

- can be easily compromised by human side situations (no self-defence);

- Vulnerable in case of direct attacks;

- Extremely difficult to adopt if network environment changes fast.

4. A novel approach in buffer overflow detection

Considering the fact that IPS/IDS systems are able to detect and prevent buffer overflow attacks and worm propagation, an additional module can be integrated into host-based IDS/IPS systems. Based on Sandbox approach such mechanism can provide an extra check of data before these data are being executed in the stack. It means that data can get the access to execute only after executing in the stack's "sandbox". In accordance with Figure 3, data that successfully passed check-execution in Stack Pre-Processor can be secure executed in stack, otherwise it will generate a violation and program or process's data will be terminated as a malicious or suspicious content.

Figure 3: Stack Pre-Processor mechanism

By putting and executing data in Stack Pre-Processor, it possible to detect and prevent buffer overflow attacks, unauthorized systems and API callings, program entry pointers, execution unsafe function ( e.g. - getc(), strncpy(), strcat() and etc.) and exploiting vulnerabilities in stack structure. In addition, this approach can provide a passive or active static analysis of ELF files that may contain exploits responsible for buffer overflow attacks. Finally, in accordance with defined rules or signatures, stack pre-processor process can block (stop, suspend, terminate) execution of data during initialization or boot process.