Chapter 13: Intrusion Detection Systems and Network Security
NIDS Tools
There are numerous examples of NIDS Tools in the marketplace, from open source projects to commercial entries. Snort - an IDS created in 1998 Suricata - an open source IDS Both of these IDSs run on Windows and Linux.
Banner Grabbing
Is a technique used to gain information about a remote server and is often used as part of a fingerprinting attack.
Automated Alerting and triggers
SIEMS have the ability through a set of rules and the use of analytical engines to identify specific predetermined patterns and either alert or react to them. Automated alerting can remove much of the time delays between specific activity and security operation reaction.
SIEM (Security Information and Event Management)
are a combination of hardware and software designed to classify and analyze security data from numerous sources.
Advanced Malware Tools
include tools such as Yara, a command-line pattern matcher to look for indicators of compromise (IOCs) in a system.
is a common mean of communication in the enterprise, and it is common to attach files to an email to provide additional information. Transferring information out of the enterprise by email is a concern for many organizations.
Time synchronization
is a common problem for computer systems. When multiple systems handle aspects of a particular transaction, having them all have a common time standard is essential if one is going to compare logs from different systems. SIEMS can handle both time readings simultaneously, using UTC for correlation across the entire enterprise, and local time for local processing.
Portscan
is a reconnaissance activity a potential attacker uses to find out information about the systems they want to attack.
Intrusion Detection System (IDS)
is a security system that detects inappropriate or malicious activity on a computer or network.
Correlation
is the connection of events based on some common basis
Active vs Passive HIDSs
on a Passive System, the HIDS watches the activity, analyzes it, and generates alarms. On a Active System (basically an IPS), the IDS can react to the activity it is analyzing. These reactions can range from something simple such as running a script to turn a process on or off, to something as complex as modifying file permissions, terminating the offending processes, logging off specific users, and reconfiguring local capabilities to prevent specific users from logging in for the next 12 hours.
Active vs Passive NIDS
on a Passive System, the NIDS watches the traffic, analyzes it, and then generates alarms. A Active NIDS (Basically a NIPS) can react to the traffic it is analyzing - It can send a TCP message to interrupt a potential attack or modify firewall rules to block specific source IP addresses. TCP Reset is the TCP message sent out by the IDS.
Switched Port Analyzer
the term Switch Port Analyzer (SPAN) is usually associated with cisco switches - other vendors refer to the same capability as Port Mirroring or Port Monitoring.
Disadvantages of an NIDS:
An NIDS has certain disadvantages: -It is ineffective when traffic is encrypted. When network traffic is encrypted form application to application or system to system, an NIDS sensor will not be able to examine that traffic. -It can't see traffic that cannot cross it. The IDS sensor can examine only traffic crossing the network link it is monitoring. -It must be able to handle high volumes of traffic. As network speeds continue to increase, the network sensors must be able to keep pace and examine the traffic as quickly as it can pass through the network. -It doesn't know about the activity occurring on the hosts themselves.
Context-Based Signatures
Are generally more complicated, as they are designed to match large patterns of activity and examine how certain types of activity fit into the other activities going on around them. Example, Match a potential intruder scanning for open web servers on a specific network. Identify a ping flood attack and Identify a Nessus scan.
Content-Based Signatures
Are generally the simplest. They are designed to examine the content of such things as network packets or log entries.
Cloud-based DLP
As data moves to the cloud, so does the need for data loss prevention. The DLP industry has responded with could based-DLP solutions designed to manage these and other could-related issues while still affording the enterprise visibility and control over data transfers.
IDS Overview
As mentioned, an IDS is somewhat like a burglar alarm. It watches the activity going around it and tries to identify undesirable activity. IDS are divided into 2 main categories: 1. Host-based IDS (HIDS) 2. Network-based IDS (NIDS) typically and IDS has the following logical components: -Traffic Collector -Analysis Engine -Signature Base -User interface and reporting
Analytics
Big data analytics is currently all the rage in the IT industry with claims of how much value can be derived from datasets. NIDS/NIPS as well as other detection equipment can certainly create large datasets, especially when connected to other data sources such as log files in a SIEM solution.
Traffic collector/sensor (IDS logical component)
Collects activity/events for the IDS to examine. On an HIDS, This could be log files, audit logs, or traffic coming to or leaving a specific system. On an NIDS, this is typically a mechanism for copying traffic off the network link - basically functioning as a sniffer (packet analyzer). This component is often referred to as a sensor.
Host based IDS (HIDS)
Examines activity on an individual system such as a mail server, web server, or individual PC. It is concerned only with an individual system and usually has no visibility into the activity on the network or systems around it.
Network based IDS (NIDS)
Examines activity on the network itself. It has visibility only into the traffic crossing the network link it is monitoring and typically has no idea of what is happening on individual systems.
Analysis Engine
Examines the collected network traffic and compares it to known patterns of suspicious or malicious activity stored in the signature database. The analysis engine is the "brains" of the IDS
Intrusion Prevention System (IPS)
An IPS (active IDS) monitors network traffic for malicious or unwanted behavior and can block, reject, or redirect that traffic in real-time. Note: An IDS is like a burglar alarm - it watches and alerts you when something bad happens. An IPS is like an armed security guard - it watches, stops the bad activity, and then lets you know what happened.
Data Loss Prevention (DLP)
refers to technology employed to detect and prevent transfers of data across an enterprise. Employed at key locations, DLP technology can scan packets for specific data patterns. This technology can be tuned to detect account numbers, secrets, specific markers, or files. When these specific elements are detected, the system can block the transfer.
Host based IDSs:
A Host based IDS examines log files, audit trails, and network traffic coming into or leaving a specific host. It can operate in Realtime, looking for activity as it occurs, or in Batch Mode, looking for activity on a periodic basis. Within the log files, the HIDS is looking for certain activities that typify hostile actions or misuse, such as the following: -Logins at odd hours -Login authentication failures -Additions of new user accounts -Modification or access of critical system files -Modification or removal of binary files (executables) -Starting or stopping processes -Privilege escalation -Use of certain programs Notes: Critical files are those that are vital to the system's operation or overall functionality. They may be program (or binary) files, files containing user accounts and passwords, or even scripts to start or stop system processes. Any unexpected modifications to these files could mean the system has been compromised or modified by an attacker. By monitoring these files, the HIDS can warn users of potentially malicious activity. Decision Trees: In IDS, a decision tree is used to help the analysis engine quickly examine traffic patterns and eliminate signatures that don't apply to the particular traffic or activity being examined, so that the fewest number of comparisons need to be made. Analyst-Driven log analysis: Log analysis is the art of translating computer-generated logs into meaningful data. Human analysts can add value through the interpretation of information in context with other sources of information.
Port Scanner
A port scanner is a tool designed to probe a system or systems for open ports/services. The good/bad news about port scanners is that the "bad guys" use them basically for the same purpose and the "good guys" use them. Port scanners can be used to do the following: -Search for "live" hosts on a network. Most port scanners enable you to perform a quick scan using ICMP, TCP, or UDP packets to search for active hosts on a given network or network segment. -Search for any open ports on the network. Port scanners are most often used to identify any open ports on a host, group of hosts, or network. -Search for specific ports. Port scanners can also be configured to just look for specific services. -Identify Services on ports. -Look for TCP/UDP services. By examining the responses from each port, you can typically deduce a bit more information about the systems you are scanning, as detailed here: -Open = Open ports accept connections -Closed = you will typically see this response when the scanned target returns an RST packet. -Filtered = you will typically see this response when an "ICMP Unreachable" error is returned. This usually indicates that the port is being filtered by a firewall or other device. -Additional types = Some port scanners will attempt to further classify responses, such as dropped, blocked, denied, timeout, etc.. A SYN scan may return different results than a NULL scan or FIN scan. You'll want to run both TCP and UDP scans as well. You may need to alter your scanning approach to use multiple techniques at different times of the day/night to ensure complete coverage.
Protocol Analyzer (Packet Sniffer)
A protocol analyzer (packet sniffer, network analyzer, network sniffer) is a piece of software or an integrated software/hardware system that can capture and decode network traffic. From a security perspective, protocol analyzers can be used for a number of activities such as: -Detecting intrusions or undesired traffic. -Capturing traffic during incident response or incident handling -Looking for evidence of botnets, trojans, and infected systems. -Looking for unusual traffic or traffic exceeding certain thresholds -Testing encryption between systems or applications. From a network administration perspective, protocol analyzers can be used for activities such as these: -Analyzing network problems -Detecting misconfigured applications or misbehaving applications -Gathering and reporting network usage and traffic statistics -Debugging Client/Server communications. Note: A sniffer must us a NIC placed in Promiscuous (promisc) mode; Otherwise, it will not see all the network traffic coming into the NIC.
Network-based IDS (NIDS)
Actually came along a few years after host based systems. Host based systems (HIDS) involved the consumption of time, energy, and cost to manage these systems. The desire for a "better way" grew along with the amount of interconnectivity between systems and the amount of malicious activity coming across the network themselves. The NIDS integrated very well into the concept of Perimeter Security. Like Host based systems, NIDS looks for certain activities that typify hostile actions or misuse, such as: -Denial Of Service (DOS) attacks -Port Scans or Sweeps -Malicious content in the data payload of a packet/packets -Vulnerability scanning -Trojans, Viruses, or Worms -Tunneling -Brute Force attacks In the simplest form, an IDS has 3 major components: 1. Traffic Collector 2. Analysis engine 3. Reports 4. A User Interface The Traffic Collector is specifically designed to pull traffic from the network (sniffer). The network Analysis Engine must be able to collect packets and examine them individually or, if necessary, reassemble them into an entire traffic session. The NIDS Signature Database is usually much larger than tat of a host-based system. Some of the signatures themselves can be quite large, as the NIDS must look at network traffic occurring in a specific order over a period of time to match a particular pattern. The Traffic Collector , Analysis Engine, and Signature Database were bundled together into a single entity called a Sensor or Appliance. The sensors would report o and be controlled by a client system or master console. Placement of the sensors very quickly became an issue for most security personnel, as the sensors obviously need to have visibility of the network traffic in order to analyze it. Because most organizations with NIDSs also have firewalls, the location of the NIDS relative to the firewall had to be considered. Placed before a firewall, the firewall would see all the traffic coming in from the internet, including attacks to the firewall itself. This includes traffic that the firewall blocks, the NIDS would overwhelm the human operators managing the system. Placed after a firewall, the NIDS sees and analyzes the traffic that is being passed through the firewall into the corporate network. This doesn't allow the NIDS to see attacks against the firewall, but it results in fewer alarms and it is the most popular placement of NIDS sensors.
Advantages of an NIDS:
An NIDS has certain advantages that make it a good choice for certain situations: -Providing IDS coverage requires fewer systems. With a few well placed NIDS sensors, you can monitor all of the network traffic going in and out of your organization. -Deployment, Maintenance, and Upgrade costs are usually lower. The fewer systems that have to be managed and maintained to provide IDS coverage, the lower the cost to operate the IDS. -An IDS has visibility into all network traffic and can correlate attacks among multiple systems. Well placed NIDS sensors can see the "big picture" when it comes to network-based attacks
Honey Pots and Honey Nets
As often is the case, one of the best tools for information security personnel has always been knowledge. To secure and defend a network and the information systems on that network properly, security personnel need to know what they are up against. Often this sort of information is passed through white papers, conferences, mailing lists or even word of mouth. Information is also gathered through examination and forensic analysis, often after a major incident has already occurred and information systems are already damaged. One of the most effective techniques for collecting this type of information is to observe this activity first hand - watching an attacker as they probe, navigate, and exploit their way through the network. A Honeypot, sometimes called a digital sandbox, is an artificial environment where attackers can be contained and observed without putting real systems at risk. Anytime an attacker has been lured into probing or attacking the virtual network, the honeypot records the activity for later analysis. Honeypots take a lot of time and effort to manage and maintain, and even more effort to sort, analyze and classify the traffic the honeypots collects. Unless they are developing security tools, most companies focus their limited security efforts on preventing attacks, and in many cases, companies aren't even concerned with detecting attacks as long as the attacks are blocked, unsuccessful, and don't affect business operations. A Honeynet is a collection of one or more Honeypots
Disadvantages of HIDS:
HIDS can also have certain disadvantages that must be weighed in making the decision of whether to deploy this type of technology: -The HIDS must have a process on every system you want to watch. You must have an HIDS process or application installed on every host you want to watch. -The HIDS can have a high cost of ownership and maintenance. Depending on the specific vendor and application, an HIDS can be fairly costly in terms of time and manpower to maintain. -The HIDS uses local system resources. To function, the HIDS must use CPU cycles and memory from the system it is trying to protect. -The HIDS has a vey focused view and cannot relate to activity around it. The HIDS can only see activity only on the host it is protecting. -The HIDS, if logging only locally, could be compromised or disabled. When an HIDS generates alarms, it typically stores the alarm information in a file or database of some sort. Notes: A security best practice is to store or make a copy of log information, especially security-related log information, on a separate system. When a system is compromised, the attacker typically hides their tracks by clearing out any log files on the compromised system. If the log files are only stored locally on the compromised system, you'll know an attacker was present (due to empty log files) but won't know what they did or when they did it.
Advantages of HIDS:
HIDS have certain advantages that make them a good choice for certain situations: -They can be very specific to an operating system and have more detailed signatures. An HIDS can be very specifically designed to run on a certain operating system or to protect certain applications. -They can reduce false positive rates. When running on a specific system, the HIDS process is more likely to be able to determine whether or not the activity being examined is malicious. -They can examine data after it has been decrypted. With security concerns constantly on the rise, many developers are starting to encrypt their network communications. -They can be very application specific. On a host level, the ID can be designed, modified, or tuned to work very well on specific applications without having to analyze or even hold signatures for other applications that are not running on that particular system. -They can determine whether or not an alarm may impact that specific system. The ability to determine whether or not a particular activity or pattern will really affect the system being protected assists greatly in reducing the number of generated alarms.
IDS Models
In addition to being divided along the host and network lines, IDSs are often classified according to the detection model they use: Anomaly or Misuse Some analysts break the IDS model even further into 4 categories, depending on how the IDS operates and detects malicious traffic ( the same models can be applied to IPS both HIPS and NIPS. 1. Behavior based 2. Signature based 3. Anomaly based 4. Heuristics
In-Band vs Out-of-Band NIDS/NIPS
In band vs Out of band NIDS/IPS is similar to the Inline vs Passive issue in the last section. In-band solutions work great for protecting network segments that have high-value systems and a limited number of traffic types - for instance, in front of a set of database servers with serious corporate data, where only types of access would be via database connections. Out-of-band systems rely on a passive sensor, or set of passive sensors, and has the ability for greater flexibility in detection across a wider range of traffic types.
Event Deduplication
In many cases, multiple records related to the same item can be generated. To avoid these issues, using a special form of correlation, where records are determined to be duplicates of a specific event, the SIEM can delete all but a single record of the event from multiple record-set. This Even Deduplication assists security analysts by reducing clutter in a dataset that can observe real events that have meaning. For this to happen, the events need a central store, something a SIEM solution provides.
User Interface and Reporting
Interfaces with the human element, providing alerts when appropriate and giving the user a means to interact with and operate the IDS.
Signature Database
Is a collection of patterns and definitions of known suspicious or malicious activity.
Misuse Detection Model
Is simpler to implement and therefore its the more popular of the two models. In a misuse detection model, the IDS looks for suspicious activity or activity that violates specific policies and then reacts as it has been programmed to do. This reaction can be an alarm, email, router configuration, or TCP reset message.
Network Security Monitoring
Is the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. Although an IDS will provide an indication of a rule being met or some other aspect, it typically provides a singular event. NSM is a process of collecting a bunch of different indicators and then using these points of data and context from which they are examined to come to a more complete understanding of what is happening. A Linux distribution specifically aimed at NSM is Security Onion, and it has a whole host of tools preconfigured. Whereas IDS is an important element in detecting bad activity on a system, NSM takes this considerably further, giving you tools and techniques that can provide greater insight into what is happening.
Anomaly Detection Model
Is the more complicated of the two. In this model, the IDS must know what "normal" behavior on the host or network being protected really is. Once the "normal" behavior baseline is established, the IDS can then go to work identifying deviations from the norm, which are further scrutinized to determine whether or not that activity is malicious.
Ping of Death
Is the sending of a large Internet Control Message Protocol (ICMP) echo packet in an attempt to crash a system.
Logs/WORM
Log files exists across a wide array of sources an have a wide range of locations and details recorded. One of the valuable elements of a SIEM solution is the collection of these disparate data sources into a standardized data structure that can be employed using database tools to create informative reports Logs are written once into this SIEM datastore, and then can be read many times by different rules and analytical engines for different decision support processes. This Write Once Read Many (WORM) times concept is commonly employed to achieve operational efficiencies, especially when working with large datasets, such as log files on large systems.
Aggregation
One of the biggest key functions of an SIEM solution is the aggregation of security information sources. In this instance, aggregation refers to the collecting of information in a central place, in a common format, to facilitate analysis and decision making.
Signatures:
One of the critical elements of any good IDS is the signature database the set of patterns the IDS uses to determine whether or not activity is potentially hostile. In general, signatures can be divided into two main groups: 1. Content-based signatures 2. Context-based signatures
Resurgence and Advancement of HIDS:
The past few years have seen a strong resurgence in the use of HIDS. With the great advances in processor power, the introduction of multicore processors, and the increased capacity of hard drives and memory systems, some of the traditional barriers to running an HIDS have been overcome. The more advanced host-based offerings, which most vendors refer to as Host Based Intrusion Prevention Systems (HIPSs), combine the following elements into a single package: -Integrated System Firewall the firewall component checks all network traffic passing into or out of the host. -Behavioral and Signature-Based IDS this is a hybrid approach, uses signatures to match well known attacks and generic patterns for catching "zero day" or unknown attacks for which no signature exists. -Application Control this allows administrators to control how applications are used on the system and whether or not new applications can be installed. -Enterprise Management some host based products are installed with an "agent" that allows them to be managed by and report back to a central server. -Malware Detection and Prevention Some HIDS/HIPS include scanning and prevention capabilities that address spyware, malware, rootkits, and other malicious software.
USB Blocking
There are numerous methods of performing USB blocking - from the extreme of physically disabling the ports, to software solutions that enable a wide range of controls.
Anomaly Based Monitoring
This model is essentially the same as the behavior-based model. The IDS is taught what "normal" traffic looks like and then looks for deviations to those "normal" patterns.
Behavior-Based Monitoring (IDS)
This model relies on a collected set of "normal behavior": what should happen on the network and is considered "normal" or "acceptable" traffic. This model can potentially detect zero-day or unpublished attacks but carries a high false positive rate as any new traffic pattern can be labeled as "suspect".
Signature Based Monitoring (IDS)
This model relies on a predefined set of patterns (called signatures). The IDS has to know what behavior is considered "bad" ahead of time before it can identify and act upon suspicious or malicious traffic.
Heuristics based Monitoring
This model uses artificial intelligence to detect intrusions and malicious traffic. A heuristic model is implemented through algorithms that help an IDS decide whether or not a traffic is malicious
Network Placement
To accommodate protocol analyzers, IDS devices, and IPS devices, most switch manufacturers support Port Mirroring or a Switched Port Analyzer (SPAN) port. Depending on the manufacturer and the hardware, a mirrored port will see all the traffic passing through a specific virtual LAN (or multiple VLANs), or all the traffic passing through other specific switch ports. Another option for traffic capture is to use a Network Tap, a hardware device that can be placed inline on a network connection and that will copy traffic passing through the tap to a second set of interfaces on the tap.
Tools
Tools are a vital part of any security professional's skill set. You may not be an "assessment professional" who spends most of their career examining networks looking for vulnerabilities, but you can use many of the same tools for internal assessment activities, tracking down infected systems, spotting inappropriate behavior, etc. Knowing the right tool for the job can be critical to performing effectively.
Active vs Passive Tools
Tools can be classified as active or passive. Active tools interact with the target system in a fashion where their use can be detectable. (Loud/Noisy) Passive tools are those that do not interact with the system in a manner that would permit detection, as in sending packets or altering traffic.
Inline Network Devices:
Two methods can be employed: 1. An Inline Sensor 2. A Passive Sensor The inline sensor is one where the data packets actually pass through the device. A failure of an inline sensor would block traffic flow. A passive sensor monitors the traffic via a copying process, so that the actual traffic doesn't flow through or depend on the sensor for connectivity. Some administrators choose to have their firewalls and IPSs fail "closed", meaning that if the devices are not functioning correctly, all traffic is stopped until those devices can be repaired. Unlike traditional IDSs, an IPS must sit inline (in the flow of traffic) to be able to interact effectively with the network traffic. Most IPSs can operate in "stealth mode" and do not require an IP address for the connections they are monitoring. As they are sitting inline, most IPSs can also offer Rate Based Monitoring to detect and mitigate a Denial Of Service attack. IPSs potential weakness is that it cannot deal with encrypted traffic. To counter this problem, some IPS vendors are including the ability to decrypt Secure Socket Layer (SSL/TLS) sessions for further inspection. To do this, some IPS solutions store copies of any protected web servers' private keys on the sensor itself.
False Positives and False Negatives
When an IDS matches a pattern and generates an alarm for benign traffic, meaning the traffic was not hostile and not a threat, This is called a False Positive. Hostile activity that does not match an IDS signature and therefore goes undetected is called a False Negative. Note: To reduce the generation of false positives, most administrators tune the IDS. "Tuning" an IDS is the process of configuring it so that it works in your specific environment-generating alarms for malicious traffic and not for traffic that is normal. Effectively tuning an IDS can result in significant reductions in False-Positive traffic.
Indicators of Compromise (IOCs)
are indicators that a system has been compromised by unauthorized activity. When a threat actor makes changes to a system - either by direct action, malware, or other exploits - forensic artifacts are left behind in the system. Although the specific list will vary based on the system and the specific threat being looked for, here is a common set of IOCs that firms should monitor for: -Unusual outbound network traffic. -Anomalies in privileged user activity -Geographical irregularities in network traffic -Account log-in red flags -Increases in database read volumes -HTML response size -Large numbers of requests for the same file -Mismatched port-application traffic, including encrypted traffic on plain ports. -Suspicious registry or system file changes -Unexpected Patching of systems -Mobile device profile changes -Bundles of data in the wrong place -Web traffic with nonhuman behavior -Signs of DDOS activity, even if temporary As with many other sophisticated systems, IOCs have developed their own internal languages, protocols, and tools. Two major, independent systems for communicating IOC information are available: 1. OpenIOC 2. STIX/TAXii/CYBOX system OpenIOC was developed by Mandiant to facilitate information of IOC data, white MITRE, under contract with the US government, developed STIX/TAXII/CYBOX. MITRE designed Structured Threat Information Expression (STIX), Trusted Automated Exchange of Indicator Information (TAXII), and Cyber Observable Expression (CYBOX) to specifically facilitate automated information sharing between organizations.