Splunk How Search Works

(How Search Works) "Index=_internal | stats count by component" Is an example of what kind of search command?

A Transforming Command *WITH* a distributable component.

(How Search Works) Distributable (Stateful Streaming) Commands run on the Search Heads because they need to know....

About the data set. They need to know what events came before, and what events came after. They cannot run on the indexers.

(How Search Works) Distributable (Stateful Streaming) Commands operate on....

At least a sub-set of the entire result set.

(How Search Works) What is in a Splunk Bucket?

Bloom Filter .tsidx Journal.gz

(How Search Works) Even if the Lexicon grows to 5 million terms, the size of the _______ ______ will not change, meaning the read time off of the disk is the same.

Bloom Filter.

(How Search Works) If you run a Transforming Command before you run a Distributed (Remote Streaming) Command in your search, then Splunk will run the Remote Streaming Command....

Centrally. That is, it won't distribute that command to the indexers. Example: "stats foo | rex bar" (It makes MORE sense to say "rex bar | stats foo" as rex gets distributed to the indexers this way).

(How Search Works) A good way to tell if you have run a transforming command is...

Did your data structure change? Did Splunk push you over from the events tab, to the Statistics tab?

(How Search Works) In regards to Streaming Commands, there are two main types. What are they?

Distributable (Remote Streaming) Centralized (Stateful Streaming)

(How Search Works) After Events are filtered based on KV pairs in a distributed search, what happens?

Distributed Commands are Applied.

(How Search Works) After the Search Head Parses the search into distributed and centralized parts, what happens?

Distributed parts of the search are sent to the indexers.

(How Search Works) What builds the bloom filter?

Each term from the lexicon is run through a set of hashing algorithms. The output of each hash sets a bit in the array to ON.

(How Search Works) Transforming Commands operate on the ______ _____ ___.

Entire Event Set.

(How Search Works) After your base search, the SPL Commands do these things:

Evaluate Transform and Format events.

(How Search Works) After Schema is applied to events that come from the indexer (Schema on the Fly) in a distributed search - what happens?

Events are filtered based on KV pairs.

(How Search Works) How are events stored?

Events are put into BUCKETS Buckets go into INDEXES as a logical grouping Indexes are written to INDEXERS.

(How Search Works) T/F The bloom filter can have false negatives.

False, it can only have false positives.

(How Search Works) T/F The bloom filter is built off of the raw data.

False, the bloom filter is built off of the Lexicon, which is part of the TSIDX

(How Search Works) In Splunk, Index means one thing T/F

False. Index is an overloaded term. It is both a logical grouping for your data (directory) and it is also a TSIDX file - Time Series Index

(How Search Works) You should always run eval before stats because this way, it runs on the indexer, and not the search heads. T/F

False. While this does distributed it across the indexers when run first, and generally seems better. You may want to run stats first to filter down the number of results eval is actually operating on. So it depends on what you want to filter and find.

(How Search Works) As events come in the are written to a ___ bucket.

HOT

(How Search Works) Where is the Lexicon?

In the TSIDX file.

(How Search Works) Where is the Postings List?

In the TSIDX file.

(How Search Works) Auto Load Balanced Forwarding is sent to:

Indexers

(How Search Works) Distributable (Remote Streaming) Commands Run on... which systems?

Indexers - distributed

(How Search Works) After distributed parts of the search are sent to the indexers, what happens next?

Indexers fetch the events from disk.

(How Search Works) What kind of data structure is the Lexicon?

It is an array. With the term, and the postings list.

(How Search Works) Why is a journal slice always exactly 128 kb?

It isn't. It is roughly 128kb. Splunk respects your event boundaries, and tries to keep the slice to this size as best it can.

(How Search Works) What does the seek address do?

It tells us where we can find the matching events in the journal.gz slices.

(How Search Works) What does the TSIDX file do?

It tells you where the word you are looking for is located in your rawdata.

(How Search Works) When you search Events are retrieved and the results move...

Linearly through SPL commands

(How Search Works) What is in the TSIDX file? (Picture)

Picture here

(How Search Works) What does the postings list contain?

Posting Value Seek Address

(How Search Works) For journal.gz what is collected and saved into slices?

Raw Data

(How Search Works) Generally speaking, your types of Splunk commands are processed in this order: ________ ________ commands --> ________ _______ commands --> ___________ commands.

Remote Streaming Stateful Streaming Transforming

(How Search Works) Transforming Commands create a ____________ data structure.

Reporting

(How Search Works) After Distributed Commands are Applied in a distributed search, what happens?

Results are sent to the search head.

(How Search Works) After Indexers fetch events from disk, what is applied to the events

Schema on the fly Schema is applied to events

(How Search Works) Bloom Filters allow us to massively speed up our ______.

Search

(How Search Works) Distributed Search is Handled by:

Search Heads

(How Search Works) Distributable (Stateful Streaming) Commands run on.. which systems?

Search Heads (Centralized)

(How Search Works) For search: "index=world name=waldo" After the value has been located in the .tsidx file, Splunk then retrieves events with "waldo" using the ____ ______ from the TSIDX.

Seek Address.

(How Search Works) Splunk's stats command has a secret hidden feature. When you run Stats, it is not a distributable command... but Stats has the ability to _____ __ _____ the operations.

Split in Half. Stats has a distributable component that CAN run on the indexers. In this case "presetats" runs on the indexers.

(How Search Works) If I search for "index=world name=waldo" What does Splunk *FIRST* do with the value "waldo"?

Splunk Hashes the value waldo, to create a bloom filter for our search.

(How Search Works) After hashing the value of the name in the search "index=world name=waldo", what does Splunk begin searching?

Splunk begins searching "world" buckets, containing the events.

(How Search Works) Why bother making all these small 128 KB slices?

Splunk can then ONLY decompress the slices that have your data, instead of decompressing the entire multi GB file.

(How Search Works) For the search "index=world name=waldo" After searching the world buckets, splunk compares....

Splunk compares the bloom filter for "waldo" to the one in each bucket.

(How Search Works) "index=_internal | eval myCurrentSize=current_size+100" Is an example of what type of search command?

Streaming Command (I don't need to know anything about the dataset that came before or after to run this eval example)

(How Search Works) The two main categories of commands in Splunk are of This type or That...

Streaming Commands or Transforming Commands

(How Search Works) In the Job Inspector, Commands that get distributed get dumped into the...

Streaming Pipeline.

(How Search Works) In the base search "index=world name=waldo" what is the FIRST thing Splunk searches on?

TIME (4 hours in this case)

(How Search Works) The Bloom filter is fast to read, vs the _____ which grows with more unique terms.

TSIDX

(How Search Works) After comparing the bloom filter in a search to the one in each bucket, Splunk then goes and locates the value in the....

TSIDX file.

(How Search Works) A logical Index is made up of many indexes and ______ ______.

TSIDX files.

(How Search Works) When you look in the Job Inspector for your search, and look at the "eventsSearch" field it is showing you...

The "Stateful & Events pipelines"

(How Search Works) When you look in the Job Inspector for your search, and look at the "reportSearch" field it is showing you...

The "Stream Report & Report Pipelines"

(How Search Works) When you look in the Job Inspector for your search, and look at the "remoteSearch" field it is showing you...

The "Streaming Pipeline"

(How Search Works) What determines whether a term is likely to exist in the TSIDX of a bucket?

The Bloom Filter

(How Search Works) If you look at the job inspector, you can look at something called the "Event Count" - What is it?

The Event Count is: How many events were left over after I performed the Scan Count, and retrieved all the events; and then compared them to the Key Value pairs, and threw out the ones that did not match.

(How Search Works) If you look at the job inspector, you can look at something called the "Scan Count" - What is it?

The Scan Count is "How many raw events did I have to fetch off of disk for your search?"

(How Search Works) Transforming Commands Operate on the entire event set and they typically run on what system?

The Search Head

(How Search Works) After the results are sent back to the search head in a distributed search, what happens?

The Search head applies centralized streaming & transforming commands, then displays the results.

(How Search Works) What does the Lexicon contain?

The Term, and the Postings List

(How Search Works) If you uncompress journal.gz, what do you see?

The raw data itself, plus a little metadata on the top.

(How Search Works) When you run a distributed Search, what is the *FIRST* thing the search head does?

The search head parses the search into distributed and centralized parts.

(How Search Works) Distributable (Remote Streaming) Commands operate on....

They operate on Individual Events

(How Search Works) Distributable (Remote Streaming) Commands can be sent out to indexers and run in parallel because....

They operate on individual events. They don't need to know anything about the event that came before, or the event that is coming next. Thus, if I run, rex, eval where, etc, it works on just that single event at a time.

(How Search Works) If I run: "index=myIndex field1=value1 | table field1, field2, field 4 | head 10000 | table field2, field4" It is bad because....

This command will send ALL the events up to the search head, as table is a transforming command. And only then will it use head to chop off the data that does not conform. If you use the fields command, it explicitly tells splunk to drop or retain fields from your results.

(How Search Works) "index=_internal | stats count by component | eval myCount=count*100" is an example of what kind of search command?

This is a streaming command *AFTER* a transforming command. (Not a good order).

(How Search Works) Once the Lexicon is built, each term is run through how many hashing algorithms?

Three.

(How Search Works) What is a TSIDX File

Time Series Index Splunk's Secret Sauce A logical Index is made of up of may indexes/TSIDX files It is how Splunk Searches for Data

(How Search Works) If your data structure has changed after searching, you have run a ________ command.

Transforming

(How Search Works) If you run a search, and Splunk switches from the Event Tab to the Statistics tab in the UI you have run a....

Transforming Command.

(How Search Works) T/F The bloom filter can have false positives.

True, it can have false positives, but not false negatives.

(How Search Works) Once Hot buckets fill up to a pre-defined size they roll to...

WARM

(How Search Works) Events are filtered based on KV Pairs AFTER Schema on the Fly is applied. Why?

You might not have the key on disk. eg: "name=waldo" Splunk does not search for the key ("name" in this case), the key is created during schema on the fly.

(How Search Works) The Bloom Filter is in ______ format.

binary

(How Search Works) Some examples of Distributable (Remote Streaming) Commands are

eval, rex, where, rename, fields....

(How Search Works) If you turn on "fast mode" in the UI, then Splunk will only retrieve the...

fields that you asked for. (kind of like using the "fields" command for you)

(How Search Works) Your base search retrieves and ______ ______.

filters events.

(How Search Works) Two Examples of Distributable (Stateful Streaming) Commands are

head, streamstats

(How Search Works) Distributable (Remote Streaming) Commands are ones that can be sent out to the ______ and run in _______.

indexers parallel.

(How Search Works) What does the postings list do?

it tells us where we can find a specific term in the values array.

(How Search Works) This file inside a bucket, on disk, is made up of many smaller compressed slices of your events

journal.gz

(How Search Works) When it comes to the raw filesystem, where are your events stored?

journal.gz

(How Search Works) The order you put your commands in your search _______.

matters

(How Search Works) When looking at your search job inspector, to see if your searches are well constructed, there are three key fields to look for. What are they?

remoteSearch eventsSearch reportSearch

(How Search Works) If you want to know if your search got sent out to the indexer, what field should you look for in the Job inspector?

reportSearch

(How Search Works) Transforming Commands Operate on the entire event set and are non-_______.

streaming.

(How Search Works) In the bloom filter regardless of the number of _____ the bit array size ______ _____.

terms remains fixed

(How Search Works) Some examples of Transforming Commands are:

transaction, stats, top, timechart

(How Search Works) How much uncompressed data makes up a slice in journal.gz?

~128kb