4 - Collecting Security Monitoring Data

What options are there for ingesting data from a unified threat management (UTM) appliance deployed on the network edge to an SIEM?

If supported, you could deploy agent software to the UTM. If an agent is not supported, you can push data to the SIEM using a protocol such as syslog. In the latter case, you will still need to use a filter to parse and normalize the logs. Most SIEMs come with filters for the major appliance platforms, but if not supported directly, you will need to configure a custom filter.

Grep Command

In Unix-like operating systems, this command invokes simple string matching or regex syntax to search text files for specific strings. This enables you to search the entire contents of a text file for a specific pattern within each line and display that pattern on the screen or dump it to another file.

Event Log Severity: Warning

Events that are not necessarily a problem but may be in the future.

Event Log Category: Forwarded Events

Events that are sent to the local host from other computers.

Event Log Severity: Error

Events that are significant problems and may result in reduced functionality.

Event Log Severity: Audit Success/Failure

Events that indicate a user or service either fulfilled or did not fulfill the system's audit policies. These are unique to the Security log.

True or false? Syslog uses a standard format for all message content.

False—syslog messages have a PRI code, header, and message structure, but the format of messages is application-specific.

syslog

For non-Windows hosts, events are usually managed by __________ This was designed to follow a client-server model and so allows for centralized collection of events from multiple sources. It also provides an open format for event logging messages, and as such has become a de facto standard for logging of events from distributed systems. For example, syslog messages can be generated by Cisco routers and switches, as well as servers and workstations, and collected in a central database for viewing and analysis. This is a TCP/IP protocol and can run on most operating systems. It usually uses UDP port 514.

Listener/collector SIEM collection

Rather than installing an agent, hosts can be configured to push updates to the SIEM server using a protocol such as syslog or Simple Network Management Protocol (SNMP). A process runs on the management server to parse and normalize each log/monitoring source.

Heuristic Analysis and Machine Learning

A basic "IF x AND (y OR z)" type of ruleset can be improved by __________. Determining whether a number of observed data points constitute an indicator and whether related indicators make up an incident depends on a good understanding of the relationships between the observables and the context in which they occur. This means the software can use techniques to determine whether a set of data points are similar enough to "IF x AND (y OR z)" that an alert should be generated anyway.

What type of visualization is most suitable for identifying traffic spikes?

A line graph is a good way of showing changes in volume over time.

ArcSight

A vendor of SIEM log management and analytics software, now owned by HP, via the affiliated company Micro Focus. As well as cybersecurity intelligence and response, one of the crucial functions of enterprise SIEMs like _______ is the ability to provide compliance reporting for legislation and regulations such as HIPAA, SOX, and PCI DSS.

Graylog

An open-source SIEM with an enterprise version focused on compliance and supporting IT operations and DevOps.

Sensor SIEM collection

As well as log data, the SIEM might collect packet captures and traffic flow data from sniffers. Often, the SIEM software can be configured in sensor mode and deployed to different points on the network. The sensor instances then forward network traffic information back to the main management instance.

Event Log Category: Security

Audit events, such as a failed log-on or access to a file being denied.

grep -i

By default, literal search strings in grep are case-sensitive. This option ignores case sensitivity.

regex: ( ... )

Defines a matching group, with a regex sequence placed within the parentheses. Each group can subsequently be referred to by \1 for the first group, \2 for the second, and so on.

Cut Command

Enables you to specify which text on a line you want to remove from your results so that they're easier for you to read. Many cut operations use the -c option, which enables you to specify which characters to cut.

Frequency-based trend analysis

Establishes a baseline for a metric, such as number of NXERROR DNS log events per hour of the day. If the frequency exceeds (or in some cases undershoots) the threshold for the baseline, then an alert is raised.

SIEM Date/Time Synchronization

Hosts might use incorrect internal clock settings, or settings that are correct for a different time zone, or record the timestamp in a non-standard way (tools.ietf.org/html/rfc3339). These issues can make it difficult to correlate events and reconstruct time sequences. Try to ensure that all logging sources be synchronized to the same time source, using Network Time Protocol (NTP), for instance. The system also needs to deal with varying time zones and daylight savings time changes consistently. If the SIEM cannot correct for these variations, one option is to ensure that all logging sources record timestamps in the UTC time zone.

Trend Analysis Metrics: Training/threat awareness education

How well-informed are staff about cyber threats? You could measure number of programs delivered or use graded assessments to evaluate knowledge levels.

Qradar

IBM's SIEM log management, analytics, and compliance reporting platform.

grep -L

Like the behavior of the -v option, in that it returns the names of files without matching lines.

SIEM deployment considerations

Log all relevant events, but not be cluttered with irrelevant data. Establish and clearly document the scope of events. Develop use cases to define exactly what you do and do not consider a threat. Have a plan about what should be done in the event that you are alerted to a threat. Establish a robust ticketing process to track all flagged events. Schedule regular threat hunting so you don't miss any important events that have escaped alerts. Provide auditors and forensics analysts with a trail of evidence to support their duties.

SIEM Secure Logging

Logging requires sufficient IT resources because it can be both disk- and network-intensive. Large organizations can generate gigabytes or even terabytes of log data every hour. Analyzing such large volumes of data requires substantial CPU and system memory resources. It is also important to configure a secure channel so that an attacker cannot tamper with the logs being sent to the SIEM. The data store itself must have the CIA triad properties of confidentiality, integrity, and availability.

regex: {}

Matches a number of times (quantifier). For example, {2} matches two times, {2,} matches two or more times, and {2-5} matches two to five times.

regex: [ ... ]

Matches a single instance of a character within the brackets. This can include literals, ranges such as [a-z], and token matches, such as [\s] (white space) or [\d] (one digit).

regex: ?

Matches once or not at all (quantifier).

regex: $

Matches the end of a line only (anchor/boundary).

Splunk

One of the market-leading big data information gathering and analysis tools. Can import machine-generated data via a connector or visibility add-on. Connectors exist for most NOS and application platforms. The data is indexed as it is retrieved and written to a data store. The historical or real-time data captured by Splunk can then be analyzed using searches, written in Search Processing Language (SPL). The results of searches can be presented using visualization tools in custom dashboards and reports, or configured as triggers for alerts and notifications. Can be installed as local enterprise software or used as a cloud solution. There is also a Light product for smaller networks and a dedicated Enterprise Security module. The security module includes pre-configured dashboards, security intelligence searches, and incident response workflows.

Alien Vault and OSSIM (Open-Source Security Information Management)

Open-Source Security Information Management (OSSIM) is a SIEM product Now owned by AT&T and is being rebranded as AT&T Cybersecurity. As well as standard SIEM functions such as asset discovery and log management, OSSIM can integrate other open-source tools, such as the Snort IDS and OpenVAS vulnerability scanner, and provide an integrated web administrative tool to manage the whole security environment.

grep -l

Returns the names of the files with matching lines rather than the lines themselves. Primarily used in multi-file grep searches.

grep -c

Returns the total count of matching lines rather than the lines themselves.

grep -v

Reverses the command's default behavior, returning only lines that do not match the given string.

SIEM Parsing and Normalization

SIEM solutions need a way of standardizing the information from these diverse sources. SIEM software features connectors or plug-ins to collect and interpret (or parse) the logs from distinct types of systems and to account for differences between vendor implementations. Usually parsing will be carried out using regular expressions tailored to each log file format to identify attributes and content that can be mapped to standard fields in the SIEM's reporting and analysis tools.

Event Log Severity: Information

Successful events.

regex: |

The OR operand (logic).

Which default port do you need to allow on any internal firewalls to allow a host to send messages by syslog to an SIEM management server?

The default port for syslog is UDP 514. If the syslog implementation is using reliable delivery, the default TCP port is 1468.

Trend Analysis Metrics: Externally measured threat levels

What is the security landscape across the Internet in general? Are there any major new threats for you to account for?

SIEM Queries

Where a correlation rule matches data as it is first ingested in the SIEM, a query extracts records from among all the data stored for review or to show as a visualization.

ELK/Elastic Stack

A collection of tools providing SIEM functionality: • Elasticsearch—The query and analytics tool. • Logstash—Log collection and normalization. • Kibana—A visualization tool. • Beats—Endpoint log collection agents.

Regular Expression (regex)

A method used in programming for pattern matching. Regular expressions provide a flexible and concise means to match strings of text. For example, a ___________________________ could be used to search through large volumes of text and change all occurrences of "cat" to "dog".

Awk

A scripting engine geared toward modifying and extracting data from files or data streams, which can be useful in preparing data for analysis. Programs and scripts run in ______ are written in the ______ programming language. The ______ keyword is followed by the pattern, the action to be performed, and the file name. The action to be performed is given within curly braces. The pattern and the action to be performed should be specified within single quotes. If the pattern is not specified, the action is performed on all input data; however, if the action is not specified, the entire line is printed.

Bash

A scripting language and command shell for Unix-like systems. It is the default shell for Linux and macOS. Tools like grep, cut, and sort are built into the ____ shell. Beyond individual command entry, ____ can run complex scripts. Like standard programming languages, ____ supports elements such as variables, loops, conditional statements, functions, and more.

SIEM Correlation Rules

A statement that matches certain conditions. These rules use logical expressions, such as AND and OR, and operators, such as == (matches), < (less than), > (greater than), and in (contains). For example, a single-user log-on failure is not a condition that should raise an alert. Multiple user log-on failures for the same account, taking place within the space of one hour, is more likely to require investigation and is a candidate for detection by a correlation rule.

Volume-based trend analysis

Can be performed with simpler indicators. For example, one simple metric for determining threat level is log volume. If logs are growing much faster than they were previously, there is a good chance that something needs investigating. Also applies to network traffic. You might also measure endpoint disk usage. Client workstations don't usually need to store data locally, so if a host's disk capacity has suddenly diminished, it could be a sign that is being used to stage data for exfiltration.

Statistical deviation analysis

Can show when a data point should be treated as suspicious. Uses the concept of mean (the sum of all values divided by the number of samples) and standard deviation. Standard deviation is a measure of how close values in the set are to the mean. If most values are close to the mean, standard deviation is low. Techniques such as regression and clustering can be used to determine whether a certain data point is not aligned with the relationships that most data points share. For example, a cluster graph might show activity by standard users and privileged users, invoking analysis of behavioral metrics of what processes each type runs, which systems they access, and so on. A data point that appears outside the two clusters for standard and administrative users might indicate some suspicious activity by that account.

Event Log Category: Application

Events generated by applications and services, such as when a service cannot start.

Event Log Category: System

Events generated by the operating system and its services, such as storage volume health checks.

Event Log Category: Setup

Events generated during the installation of Windows.

regex: +

Matches one or more occurrences (quantifier). A quantifier is placed after the term to match; for example, \s+ matches one or more white space characters.

regex: ^

Matches the start of a line only (anchor/boundary).

regex: *

Matches zero or more times (quantifier).

Behavior-based detection

Means that the engine is trained to recognize baseline traffic or expected events associated with a user account or network device. Anything that deviates from this baseline (outside a defined level of tolerance) generates an alert. The engine does not keep a record of everything that has happened and then try to match new traffic to a precise record of what has gone before. It uses heuristics to generate a statistical model of what the baseline looks like. It may develop several profiles to model behavior at various times of the day. This means that the system generates false positive and false negatives until it has had time to improve its statistical model of what is normal.

Head Command

Outputs the first 10 lines respectively of a file you provide. You can also adjust this default value to output more or fewer lines.

Tail Command

Outputs the last 10 lines respectively of a file you provide. You can also adjust this default value to output more or fewer lines. This tool is useful for reviewing the most recent entries in a log file.

A SIEM will help with most of the regular duties involved in staffing a SOC or CSIRT, such as:

Perform triage on alerts, escalating true positives to incident response and dismissing false positives. Review security data sources to check that log collection and information feeds are functioning as expected. Review CTI to identify priorities or potential impacts from events occurring at other companies and all over the Internet. Perform vulnerability scanning and management. Identify opportunities for threat hunting, based on CTI and overall alert and incident status.

Security Information and Event Management (SIEM)

Solutions provide real-time or near-real-time analysis of logs and alerts generated by network hardware and applications. Used to provide expanded insights into intrusion detection and prevention through the aggregation and correlation of security intelligence. Solutions can be implemented as software, hardware appliances, or outsourced managed services.

Working with the same data file, write the command to show only the lines where the destination IP address is 10.1.0.10 and the destination port is 21.

The easiest way to do this is to pipe the result from one grep command into another. Remember that you need to escape characters such as periods and commas. The $ operator in the second command is a handy way of selecting the destination port, which is at the end of the line. grep "10\.1\.0\.10\," iptables-csv | grep "21$" Note that you need to include the comma delimiter to avoid selecting partial source IP addresses. The regex "10\.1\.0\.10" would match 10.1.0.102.

You need to analyze the destination IP address and port number from some firewall data. The data in the iptables file is in the following format: DATE,FACILITY,CHAIN,IN,SRC,DST,LEN,TOS,PREC,TTL,ID, PROTO,SPT,DPT Jan 11 05:33:59,lx1 kernel: iptables,INPUT,eth0,10.1.0.102,10.1.0.1,52, 0x00,0x00,128,2242,TCP,2564,21 Write the command to select only the necessary data, and sort it by destination port number.

The following command selects columns 6 (destination IP address) and 14 (destination port) and then sorts by column 2—remember that the piped output to the sort command has two columns, not 14. The -n switch makes the sort order numeric rather than alphabetical. cut -d "," -f6,14 iptables | sort -t "," -k2 -n

Piping |

The output of a command can be used as the input for another command—a process called piping. Using the __________ causes the following command to take the output of a previous command as its input. For example, to return only lines in syslog. txt that deal with the NetworkManager process, while also cutting each line so that only the date, time, source, and process display, you would enter: grep "NetworkManager" /var/log/syslog | cut -d " " -f1-5 | sort -t " " -k3

Anomaly Analysis

The process of defining an expected outcome or pattern to events, and then identifying any events that do not follow these patterns. This is useful in tools and environments that enable you to set rules. If network traffic or host-based events do not conform to the rules, then the system will see this as an anomalous event. For example, the engine may check packet headers or the exchange of packets in a session against RFC standards and generate an alert if they deviate from strict RFC compliance. This is useful because you don't need to rely on known malicious signatures to identify something unwanted in your organization, as this can lead to false negatives.

Trend Analysis

The process of detecting patterns within a dataset over time and using those patterns to make predictions about future events. Applied to security intelligence, _________ can help you to judge that specific events over time are related and possibly indicate that an attack is imminent. It can also help you avoid unforeseen negative effects that result from an attack if you can't stop the attack altogether. Also enables you to review past events through a new lens. For example, when an incident happens, you'll usually attribute it to one cause. However, after time has passed and you gather more intelligence, you may gain a new perspective and realize that the nature of the cause is different than you had originally thought. A trend is difficult to spot by examining each event in a log file. Instead, you need software to visualize the incidence of types of event and show how the number or frequency of those events changes over time.

Trend Analysis Metrics: Number of alerts and incidents and detection/response times

These types of metrics show how well security operations are performing. You could potentially also measure hours lost or impact in cost terms, though these things are hard to measure and quantify.

grep -w

Treats literal search strings as discrete words. By default, the string add will also return address. With this option, the string add will only return instances of the word add by itself.

Windows PowerShell

Used to manage both local and remote hosts. Offers much greater functionality than the traditional Windows command prompt. Functions mainly through the use of cmdlets, which are specialized .NET commands. These cmdlets typically take the syntax of Verb-Noun, such as Set-Date, to change a system's date and time. Like other command shells, the cmdlet will take whatever valid argument the user provides. Also able to execute scripts written to its language.

Trend Analysis Metrics: Compliance

What percentage of compliance targets are being met? Is the percentage going up or down? If going down, is this because the compliance targets are increasing or getting tougher to meet, or because policies are not being followed correctly?

Agent-based SIEM collection

With this approach, you must install an agent service on each host. As events occur on the host, logging data is filtered, aggregated, and normalized at the host, then sent to the SIEM server for analysis and storage. Agents could be configured to forward event and application logs, such as the Elastic Stacks Beats agents (elastic.co/products/beats), or intrusion detection data, such as OSSEC (ossec.net).

Trend Analysis Metrics: Network and host metrics

You can measure any number of network metrics (volume of internal and external traffic, numbers of log-ons/log-on failures, number of active ports, number of authorized or unauthorized devices, instances of unauthorized software, creation of administrative accounts, and so on) but they might only be interesting from a security perspective if they can reveal deviations from the network baseline. Most networks change considerably over a period for genuine business reasons

Which two factors do you need to account for when correlating an event timeline using an SIEM?

You need to validate that all log sources were synchronized to the same time source. You need to account for any variations in time zone for the different sources.