Splunk 7 - System Admin

Splunk preconfigured indexes

- _internal - _ audit - _introspection - _thefishbucket - summary - main

Installation: (as account running Splunk)

-*NIX -un-compress the .tar.gzfile in the path you want Splunk to run from ---Also available as rpm, deb -Windows-execute the .msi installer and follow the wizard steps

Hardware Recommendation Specs - Indexer

--OS - Linux or Windows 64-bit distribution --Network - 1Gb Ethernet NIC (Optional second NIC for a management network) --Memory - 12 GB RAM --CPU - Intel 64-bit chip architecture, 12 CPU cores Running at 2+ GHz --Disk - Disk subsystem capable of 800 IOPS RAID 10

Hardware Recommendation Specs - Search Head

--OS - Linux or Windows 64-bit distribution --Network - 1Gb Ethernet NIC (Optional second NIC for a management network) --Memory - 12 GB RAM --CPU - Intel 64-bit chip architecture, 4 CPUs, quad-core per CPU, Running at 2+ GHz --Disk -2 x 10K RPM 300GB SAS drives -RAID 1

Why Create Indexes

-Access Control -Retention

Search time

-App/User context -User-related activity, such as searching

Splunk Web

-Browser-based user interface - Provides both a search and management front end for the splunkd process -Runs on port 8000 by default -http://<server_name>:<port>

Monitoring Console (MC)

-Collects data about itself -Splunk admin-only app -Monitors performance, resource usage, and more

An app is a collection of:

-Configuration files -Scripts, web assets, etc.

The system admin can:

-Create new indexes -Control which indexes users can access

Free license

-Disables alerts, authentication, clustering, distributed search, summarization, and forwarding to non-Splunk servers -Allows 500mb/day of indexing and forwarding to other Splunk instances

Enterprise trial license

-Downloads with product -Features same as Enterprise except for 500mb per day limit -Only valid for 60 days, after which one of the other 3 license types must be activated

Create a user account that is used to run Splunk:

-For input, Splunk must be able to access data sources On *NIX, /var/log is not typically open to non-root accounts -On *NIX, non-root accounts cannot access ports < 1024 -On Windows Use a domain account if Splunk has to connect to other servers Otherwise, use a local machine account that can run services - Make sure the Splunk account can access scripts used for inputs and alerts

Forwarders

-Forwarders collect data and send it to Splunk servers -Install forwarders at data source (usually production servers)

Server Settings Menu

-General settings -Login background -Email settings -Server logging -Deployment client -Search preferences

Index time

-Global context -User-independent and background tasks such as inputs, parsing, indexing, etc.

Phases and processes - Universal Forwarder

-Input (inputs.conf) -Forward (outputs.conf) -Parse (props.conf [only very limited cases])

Phases and processes - Search Head

-Input (inputs.conf) -Forward (outputs.conf) -Parser (props.conf) [Search time parsing] -License (License slave/License Master) -Search (Search Head)

Phases and processes - Heavy Forwarder

-Input (inputs.conf) -Forward (outputs.conf) / Forwarder & Receiver -Parse (props.conf) -License (License Slave)

Phases and processes - Indexer

-Input (inputs.conf) -Forward (outputs.conf) / Receiver -Parse (props.conf) -License (License Slave) -Search (Search Peer)

The Splunk System Administrator is primarily responsible for system management efforts which include:

-Install, configure, and manage Splunk components -Install and manage Splunk apps -Manage Splunk licensing -Manage Splunk indexes -Manage Splunk users and authentication -Manage Splunk configuration files -Monitor MC and respond to system health alerts

Splunk can be distributed and scaled in a variety of ways

-More indexers to handle more inputs -More indexers AND search heads to handle more searching

Enterprise license

-Purchased from Splunk -Full functionality for indexing, search head, deployment server, etc. -Sets the daily indexing volume -No-enforcement license, allows users to keep searching even if you are in a license violation period

What does not count against your license daily quota?

-Replicated data (Index Clusters) -Summary indexes -Splunk internal logs (_internal, _audit, etc. indexes) -Structural components of an index (metadata, tsidx, etc.)

All .conf files have documentation and examples:

-SPLUNK_HOME/etc/system/README *.conf.spec *.conf.example

Splunk stores the input data as events in indexes in:

-SPLUNK_HOME/var/lib/splunk -Set in Settings > Server Settings > General Settings -Can override on a per-index basis

Splunk Core Components and Processes

-Searching -Indexing -Parsing -Inputs

Forwarder license

-Sets the server up as a heavy forwarder -Applies to non-indexing forwarders -Allows authentication, but no indexing

splunk btool conf-name list [options]

-Shows on-disk configuration for requested file -Run splunk btool check each time Splunk starts -Useful for checking the configuration scope and permission rules Use --debug to display the exact .conf file location Add --user= <user> --app=<app> to see the user/app context layering

Basic Splunk server

-Similar to server in standalone configuration -Manage deployment of forwarder configurations

Splunk can be deployed in a what kinds of configurations?

-Single Server -Distributed Infrastructure

Metrics Indexing

-Special index format for metrics and certain types of log collection

Run Splunk at Boot : *NIX

-Splunk on *NIX does not auto-start at boot time (default) -Required practice to enable boot-start, run as root: # ./splunk enable boot-start -user splunker -This modifies the *NIX boot-up configuration --Modifies /etc/init.d depending on your *NIX flavor -Pass the -user parameter to start Splunk as the correct user

Metrics Indexing supports these protocols:

-StatsD extension over UCP/TCP -Plain StatsD extension with dimension over UDP/TCP -Collectd over HTTPs using HEC

Universal Forward software includes:

-Universal Forwarder -Deployment Client

The Splunk Data Administrator is primarily responsible for data onboarding and management efforts which include:

-Work with users requesting new data sources -Document existing and newly ingested data sources -Design and manage inputs for UFs/HFs to capture data -Manage parsing, event line breaking, timestamp extraction -Move configuration through non-production testing as required -Deploy changes to production -Manage Splunk configuration files

Installing an App From a File via CLI

-splunk install app path-to-appfile -Extract the app in the proper location. cd SPLUNK_HOME/etc/apps tar -xf path-to-appfile -Apps may require a splunkd restart

To delete an app:

-splunk remove app <app_folder> -Or, navigate to SPLUNK_HOME/etc/apps and delete the app's folder and all its contents -Restart the Splunk server -Safer to disable it or move the app's files to another location (/splunk disable app [app_name] -auth <username>:<password>)

Index Time Merging of Configurations, If there are conflicts:

-the setting with the highest precedence is used -local always takes precedence over default

Changes made by editing .conf files are......

......not automatically detected -To force reload, go to: http://servername:webport/debug/refresh -Reloads many of the configurations, including inputs.conf, but not all -To reload all configurations, restart Splunk - Splunk Web: Settings> Server controls> Restart Splunk - CLI: splunk restart

Changes made using Splunk Web or the CLI......

.....may not require restart. -A message appears if restart is required (i.e. changing server settings)

If the indexing exceeds the allocated daily quota in a pool,.....

...an alert is raised in Messages (pool warning) on any page in Splunk Web

SPLUNK_HOME Directory (/opt/splunk)

/bin (executables) /etc (licenses,configs) /var

/var directory

/lib/splunk (indexes) directory

/etc/apps/ directory

/search /launcher /<custom app>

/etc directory

/system /apps /users

Index-Time Precedence Order

1. etc/system/local 2. etc/apps/search/local 3. etc/apps/unix/local 4. etc/apps/search/default 5. etc/apps/unix/default 6. etc/system/default

Metrics data counts against a license at a fixed _____ bytes per metric event

150

______ or more warnings on an enforced Enterprise license or ___ warnings on a Free license, in a rolling ____-day period, is a violation

5, 3, 30

Scenario: What are the /var/log/secure.log input configurations and where are they specified?

> splunk btool inputs list monitor:///var/log/secure.log --debug etc/apps/search/local/inputs.conf etc/system/local/inputs.conf etc/system/default/inputs.confi etc/apps/search/local/inputs.conf [monitor:///var/log/secure.log] host = myIndexer index = default sourcetype = linux_secure

Configuration changes are saved in .conf files under:

SPLUNK_HOME/etc/

Apps are installed under:

SPLUNK_HOME/etc/apps

Licenses are stored under...

SPLUNK_HOME/etc/licenses

Avoid storing configurations in ____________________

SPLUNK_HOME/etc/system

ex. default directories:

SPLUNK_HOME/etc/system/default SPLUNK_HOME/etc/apps/search/default

Manage your configurations in the ____________________ app's local directory

Searching and Reporting apps' local directory --(SPLUNK_HOME/etc/apps/search/local) unless the configuration is only for a specific app

Splunk Enterprise software package includes:

Splunk Enterprise: -Indexer(Search Peer) -Search Head -Deployment Server -License Master -Heavy Forwarder -Master Node

You can change settings using:

Splunk Web, CLI, SDK, app install, and/or direct edit

_internal index

Splunk indexes its own logs and metrics from its processing here

_audit index

Splunk stores its audit trails and other optional auditing information

After installation, Windows forwarders:

Starts automatically

ex.> etc/system/local/inputs.conf

[monitor:///var/log/secure.log] host=myIndexer

ex.> etc/apps/search/local/inputs.conf

[monitor:///var/log/secure.log] sourcetype=linux_secure host=webserver

Correct method to override settings in default directories is to:

add or change the setting in the local directory at the same scope

-An event index ______________ be converted into a metrics index (or vice-versa)

cannot

_thefishbucket index

contains checkpoint information for file monitoring inputs

main index

default index for inputs, located in the defaultdb directory

summary index

default index for summary indexing system

If two or more apps at the same level of precedence have conflicts between them, the conflicts are resolved in ___________________ order by app directory name.

lexicographical (ASCII)

.conf files --Only modify configurations to files in a __________ directory

local

The daily license quota resets at

midnight

After installation, *NIX forwarders:

must be manually started on *NIX until boot-start is enabled

Event-based indexes are ______ the same as metrics-based indexes

not

Adding a license via CLI

splunk add licenses <path_to_file>

btool examples

splunk help btool splunk btool check splunk btool inputs list splunk btool inputs list monitor:///var/log splunk btool inputs list monitor:///var/log --debug

Splunk Default Ports

splunkd: 8089 Universal forwarder: 8089 Splunk forwarder receiver (Indexing): 9997 Splunk forwarder receiver, encrypted (Indexing): 9998 Splunk Indexer Clustering: 8080 Splunk Web: 8000 Web app-server proxy: 8065 KV Store: 8191

Estimating Index Growth Rate

•Splunk compresses the event's raw data as it is indexed -Indexing components are added to each bucket -If events have many searchable terms, the index components are larger -If the data contains fewer searchable terms and less variety, the index is smaller

Inputs

•Splunk instances that monitor configured inputs and forward the data to the index •Best practice data collection method •Requires minimal resources and typically installed on the machines that produce the data

Freezing: Data Expiration

•The oldest bucket is deleted from an index when: -The index's maximum size is reached -The bucket's age exceeds the retention time limit -All the events in the bucket have expired •Splunk will never exceed the maximum overall size of an index -Buckets may be deleted even if they have not reached the time limit •You can optionally configure the frozen path -Splunk moves the bucket's raw data to this location before deletion -Frozen buckets are not searchable •Frozen data can be brought back (thawed) into Splunk if needed

Linux Setting Recommendations 2

•Turn Transparent Huge Pages (THP) off on Splunk Enterprise servers docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/SplunkandTHP

Warm and Cold Bucket Names

•Warm bucket names identify the time range of the events contained in that bucket •When a warm bucket rolls to cold, the entire bucket is moved, maintaining its name •At search time, Splunk scans the time range on a bucket name to determine whether or not to open the bucket and search its events

Start-up Account Best Practice:

Do not run Splunk as super-user - For example, root on *NIX, administrator on Windows

Calculating Index Storage

Limiting size on disk is the most common method of controlling index growth

Standalone Deployment

•Single server -All functions in a single instance of Splunk -For testing, proof of concept, personal use, and learning -This is what you get when you download Splunk and install with default settings

Estimating Index Growth Rate Best Practice:

Get a good growth estimate. -Input your data in a test/dev environment over a sample period -If possible, index more than one bucket of events -Examine the size of the index's db directory compared to the input -MC: Indexing > Indexes and Volumes > Index Detail: Instance

Standalone Deployment Recommendation

Have at least one test/development setup at your site

-Only add the items you are overriding — do not make a copy of the entire configuration file

To disable a default attribute TRANSFORMS for [syslog]:

_introspection index

tracks system performance and Splunk resource usage data

Hot Buckets

•After data is read and parsed, it goes through the license meter and the resulting event is written into a hot bucket •When hot buckets reach their max size or time span, they are closed and converted to warm status -Hot buckets also roll to warm automatically when the indexer is restarted -Hot and warm buckets are stored in the db directory for the index -Hot buckets are renamed when rolled to warm

What Counts As Daily License Quota?

•All data from all sources that is indexed -It is the data (full size) that flows through the parsing pipeline,per day, per indexer -It is not the amount of storage used by the indexes

Searching

•Allow users to submit search requests using SPL •Distribute search requests to the Indexers •Consolidate results and render visualizations of results •Search-time knowledge objects are stored on the search heads Examples include: field extractions, alerts, and dashboards

Buckets

•An index stores events in buckets •A bucket is a directory containing a set of raw data and indexing data •Buckets have a maximum data size and a time span -Both can be configured

Data Flow Through an Index

•Hot: Newest buckets -open for writes (readable) •Warm: Recent data, buckets are closed (read only) •Cold: Oldest data still in the index (read only) •Frozen: Deletion is the default action. If a frozen path is defined, then the bucket's raw data is archived -not searchable

Linux Setting Recommendations 1

•Increase ulimit settings -The following OS parameters need to be increased to allow for a large number of buckets/forwarders/users docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/ulimitErrors

Indexing Parsing

•Reside on dedicated machines •Receive, index, and store incoming data from forwarders •Search data in response to requests received from the search heads

splunkd

•Runs on port 8089 (default) using SSL •Spawns and controls Splunk child processes (helpers) -Splunk Web proxy, KV store, and Introspection services -Each search, scripted input, or scripted alert •Accesses, processes, and indexes incoming data •Handles all search requests and returns results