Data Science

Considering Anomaly distributions

This is not necessarily true but certain algorithms like KDE and OC SVM might make that assumption ahead of time. This ca be mitigated by a Semi Supervised or supervised anomaly detection approaches, or modeling anomalies before hand.

Latent Dirichlet Allocation

Topic Modeling where we categorizes probabilities of a variable or document ends up in an area or topic of n dimensional topics. The # of Topics is a hyper parameter that can be set

Factorization Machines

Supervised Learning Model High Dimensional and Sparse data to predict missing missing entries in matrix. matrix is var1 by var2 for classification, Recommenders and regression

Top2Vec

Takes the concept of Semantic Embedding = Documents and Words sharing a Semantic space allowing them to be compared) and Finds the centroid of densely located Document vectors using the closet word representing the centroid aka Topic Vector The Space produces Topic , Document and Word Vectors Query Word V -> Topics V -> Document V

Association Rule Mining, Support

The Fraction of support count / all existing sets even repeats aka transactions s({X,Y})/|transactions|

The Concentration/Cluster Assumption

The Region where normal data lives can be bounded basically there is a threshold of normality

Association Rule Mining, Closure

The closure of a set, cl(S), can be defined as the intersection of all closed sets containing S cl(S)=⋂ { C∣C closed , S⊆C} so all closures are closed sets because they are subsets of other closed sets

anomaly Scale

The features may define one concept or represent multiple contexts

objective function

The function being maximized or minimized in Linear Programming

Density Level Set Estimation

The goal of the density level set estimation is to generate an estimate Gb of the level set based on the n observations {X1...Xn}, such that the error between the estimator Gb and the target set G , as assessed by some performance measure which gauges the closeness of the two sets, is small.

Kernel Density Estimation (KDE)

Where Histograms are bricks , KDE are piles of sand Model of intensity of a spatial process, particularly useful at unobserved locations. It is a non-parametric way to estimate the probability density function of a random variable. based on either being distribution-free or having a specified distribution but with the distribution's parameters unspecified. We often assume our data comes from a normal distribution, or at the very least from a distribution with mean μ and variance σ^2. Nonparametric methods are not based on such parameters. Kernel density estimation is a fundamental data smoothing problem where inferences about the population are made, based on a finite data sample. is a common technique used to estimate probability density functions (PDFs). For each data point a kernel is assigned and the sum of the kernels forms the density l under the total kernels https://towardsdatascience.com/histograms-vs-kdes-explained-ed62e7753f12

Triangle Inequality Theorem

The sum of the lengths of any two sides of a triangle is greater than the length of the third side.

Prototype Assumption

There are a finite set of prototypical elements in data space that characterize the data

Density Estimation and Probabilistic Anomaly Detection Models

These Models predict anomalies by estimating normal data probability distribution a) Classic Density Estimation i) Energy Based Models b) Deep Statistical Models i) Neural Generative Models (VAE & GANS) Seem generally based on negative Log Likelihood anomaly scores ~ continuous ~ Ranking

One Class Classification Detection Anomaly Models

This discriminative approach avoids a full estimation of density by focusing on detecting the boundaries od the dense areas that has low errors when evaluating new data. This generally provides a binary density Level Set detector. better at detecting what isn't normal, and efficient sample wise A) Support Vector Data Discriminator B) Kernel based OCC C) Deep SVDD D) Deep OC-SVM

Pointwise Mutual Information (PMI)

is a measure of association of two variables. It can take positive or negative values, but is zero if X and Y are independent. Pointwise mutual information measure is not confined to the [0,1] range. Positive means that the two events co-occurring in a frequency higher than what we would expect if they would be independent event. if p(x|y)/p(y) smaller than 1, the log is negative p(x|y) < p(y) means we see more of Y=y than we see x given that Y=y. The pointwise mutual information represents a quantified measure for how much more- or less likely we are to see the two events co-occur, given their individual probabilities, and relative to the case where the two are completely independent.

Minkowski Distance

is a metric in a normed vector space which can be considered as a generalization of both the Euclidean distance and the Manhattan distance. ( |y-y|^p)^1/p

Skip Grams

is actually the opposite of CBOW: instead of predicting one word each time, we use 1 word to predict all surrounding words ("context"). Skip gram is much slower than CBOW, but considered more accurate with infrequent words.

Manifold learning (Nonlinear)

is an approach to non-linear dimensionality reduction. Algorithms for this task are based on the idea that the dimensionality of many data sets is only artificially high.

Association Rule Mining, Minimum confidence

is an arbitrary Threshold used to find an association rule since all rules have a confidence they must satisfy confidence ≥ minconf threshold

Association Rule Mining, Minimum Support

is an arbitrary Threshold used to find an association rule since all rules have a support they must satisfy support ≥ minsup threshold

Generative Adversarial Networks (GANs)

is an unsupervised learning task in machine learning that involves automatically discovering and learning the regularities or patterns in input data in such a way that the model can be used to generate or output new examples that plausibly could have been drawn from the original dataset. The generative model frames the problem as a supervised learning problem with two sub-models: A) The generator model that we train to generate new examples. takes a fixed-length random vector as input and generates a sample in the domain. The vector is drawn from randomly from a Gaussian distribution, and the vector is used to seed the generative process. B) The discriminator model that tries to classify examples as either real (from the domain) or fake (generated). takes an example from the domain as input (real or generated) and predicts a binary class label of real or fake (generated). The real example comes from the training dataset. The generated examples are output by the generator model. The two models are trained together in a zero-sum game, adversarial, until the discriminator model is fooled about half the time, meaning the generator model is generating plausible examples. https://towardsdatascience.com/introduction-to-normalizing-flows-d002af262a4b

Information Gain

is the Difference between the Entropy of Set and Entropy of a Subset. This is a heuristic metric used for Optimization of determining maximum Information Gain. The Greater the Gain the more opportunities to split data

Association Rule Mining, Confidence

is the the fraction of a itemset {X,Y} / itemset predicate X in X -> Y c(X -> Y) = s( {X,Y})/s( {X}) it is an indication of how often the rule has been found to be true. c(ABC-> D) > c(AB-> CD) >= c(A -> BCD)

doc2Vec Embeddings

is to create a numeric representation of a document, regardless of its length. it is a small extension to the CBOW model. But instead of using just words to predict the next word, we also added another feature vector, which is document-unique. the document vector intends to represent the concept of a document. It can learn document vectors and word vectors. Where words can be compared to documents and represent the doc = Semantic Embedding Paragraph Vec w Dist. Mem. (DM) Dist. Bag of Words (DBOW)

Jaccard Distance

measures dissimilarity between sample sets, commonly used to calculate an n × n matrix for clustering and multidimensional scaling of n sample sets

Gini Index

measures income inequality. A higher number means more inequality. "measure how often a randomly chosen element from the set would be incorrectly labeled" 0 = never miss labeled .5 = %50

Jaccard Similarity

measures set similarity between sets

Divisive Clustering

methods separate n objects successively into finer groupings. Starts with one all-inclusive and splits until each cluster has one or K clusters achieved

Clustering

minimize Intra-Cluster. Max Inter-Cluster Dist.summarize/utility, understanding, compression by grouping. Image/pattern Recognition Cluster Analysis: divides into clusters Clustering: are a collection of clusters Cluster: similar group

discriminative models

models capture the conditional probability p(Y | X). the model ignores the question of whether a given instance is likely, and just tells you how likely a label is to apply to the instance. it must choose or make a decision as to what class a given example belongs.

KNN

simple and efficient application of distance based classification examines each pixel to be classified, then identifies the k nearest training samples as measured in multispectral data space Expensive Testing Phase susceptible to skewed data, data can't be to similar Relationships between features and the target class are numerous, complicated and difficult to understand. Another way to describe it is when the concept is difficult to define, but you know it when you see it, then nearest neighbor might be appropriate. On the other hand, if there is no clear distinction among the groups, then the algorithm may not well suited for identifying the boundaries.

Alternative Hypothesis (H1)

states that there is a change, a difference, or a relationship for the general population. In the context of an experiment, H1 predicts that the independent variable (treatment) does have an effect on the dependent variable. the opposing statement to a null Hypothesis

Mode

the most frequently occurring score(s) in a distribution

curse of dimensionality

the phenomena that occur when High # of attributes are being classified, organized, and analyzed and the high dimensional data that does not occur in low dimensional spaces, specifically the issue of data sparsity and "closeness" of data. number of values you need to fill space is exponentially large. n ≈ (10-30) dim. Small bad quality

Data Mining

the process of analyzing data to extract information not offered by the raw data alone (Bottom Up Data is start) Techniques Clustering, Classification, Anomaly det., Association Mining used for Statistics, Math, Advanced Computing, Visualization Tasks: 1. Prediction Methods-find future values 2. Description Methods-find patterns that describe data

Anomaly Detection

the process of identifying rare or unexpected items or events in a data set that do not conform to other items in the data set. It hinges on an ability to accurately analyze time series data in real time. Time series data anomaly detection must 1) create a baseline for normal behavior in primary key performance indicators (KPIs)

Standarization

the process of transforming a variable to one with a mean of 0 and a standard deviation of 1.

Resolution

the scale/magnification needed to identify something

Null Hypothesis (H0)

the statistical hypothesis tested by the statistical procedure; usually a hypothesis of no difference or no relationship. This is an example of that statement: You are trying to determine if you believe a statement to be true or false. The null hypothesis is, in a way, the default statement, as it is presumably true and it is the test's job to challenge this

Anomaly Threshold

thresholds to detect anomalies in performance measures on devices, links, and interfaces, and display these anomalies in the Device Dashboard.

Time Series - Walk Forward Validation

training of statistical models are not time consuming, walk-forward validation is the most preferred solution to get most accurate results

Dot Product of Vectors

u1v1 + u2v2 a · b = ax × bx + ay × by + az × bz a · b = |a| × |b| × cos(90°) =0 |v| = √(v1^2 + v2^2 + v3^2) The Dot Product gives a scalar (ordinary number) answer, and is sometimes called the scalar product.

Random Cut Forests (RCF)

updates are better served with simpler choices of partition. More advanced algorithms in the inference phase can compensate for simpler updates. Unsupervised numeric high Dimensional data

Word2Vec

word2vec (CBOW +Skip Grams) is an algorithm and tool to learn word embeddings by trying to predict the context of words in a document. The resulting word vectors have some interesting properties, for example vector('queen') ~= vector('king') - vector('man') + vector('woman'). Two different objectives can be used to learn these embeddings: The Skip-Gram objective tries to predict a context from on a word, and the CBOW objective tries to predict a word from its context. Word2vec relies only on local information of language. That is, the semantics learnt for a given word, is only affected by the surrounding words. does very well in analogy tasks Such representations, encapsulate different relations between words, like synonyms, antonyms, or analogies, build some model using words, simply labeling/one-hot encoding them is a plausible way to go

Distributional Hypothesis

words that occur in the same contexts tend to have similar meanings (Harris, 1954)

Histogram Estimators

y= Bin membership size based clusters https://towardsdatascience.com/histograms-vs-kdes-explained-ed62e7753f12

Data Quality Issues

•Missing Values Ex Survey questions skipped, insufficient choices •Noise & Outliers identification •Wrong Data •Duplicate Data

Data Types and their operations

•Nominal ex ID#, Sex, color Mode, Frequency, Correlation, =, != •Ordinal ex grades Sequence (monatomic preserved)no diff or sum Rank Correlation, Percentiles, Median g=f(x) , <, > •Interval temps., Dates no meaning of zero, Median, Std. Dev. T-test , F-test, Pearson's Correlation g=a*f(x)+i -, + •Ratio Kelvin, age,Time, Counts Location of zero is defined and fixedgeometric Mean, harmonic mean, % variation g=a*f(x) * , /

Data Mining Challenges

•Scalability •Dimensionality - Computational complexity required to measure or analyze huge •Computational complexity required to measure or analyze huge •Heterogeneous & Complex data of varying types Unstructured. clusters vary in size, height, shape •Data Ownership & Distribution distributed nature too hard. need to gather or work the data wholly •Nontraditional Analysis - difficult to determine all the potential hypotheses that can be present in the data

Manifold

a collection of points forming a certain kind of set, such as those of a topologically closed surface or an analog of this in three or more dimensions.

standard deviation

a computed measure of how much scores vary around the mean score. a quantity calculated to indicate the extent of deviation for a group as a whole. 1 std = + or - 34.1% => 68.2% around mean 2 std = + or - 47.7% => 95.4% around mean 3 std = + or - 49.9% => 99.8% around mean 4 std = + or - 50% => 100% around mean

Principal Component Analysis (PCA)

a dimension-reduction tool that can be used to reduce a large set of variables to a small set that still contains most of the information in the large set. Classically useful in Anomaly detection

latent semantic analysis (LSA)

a mathematical procedure for automatically extracting and representing the meanings of propositions expressed in a text. a technique of analyzing relationships between a set of documents and the terms they contain by producing a set of concepts related to the documents and terms. A matrix containing word counts per document (rows represent unique words and columns represent each document) is constructed from a large piece of text and a mathematical technique called singular value decomposition (SVD) is used to reduce the number of rows while preserving the similarity structure among columns.

Ranking

a position on a scale that shows how good someone or something is when compared with others. similarity varies based on distance measure used

distributed representation

a representation in which information is coded as a pattern of activation distributed across many different nodes. A collection of distributed neurons where each contributes to the representation of a concept

Bag-of-words (BOW)

a simplifying representation as the bag (multiset) of its words, commonly used in methods of document classification where the (frequency of) occurrence of each word is used as a feature that disregards grammar and even word order but keeping multiplicity. loses many subtleties of a possible good representation, e.g consideration of word ordering.

Anomaly Based Objectives

a) Loss based Learning b) Distance based learning (i.e.Nearest Neighbor, Isolation Forest, Local Outlier Factor)

Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP)

an algorithm for dimension reduction based on manifold learning techniques and ideas from topological data analysis. It provides a very general framework for approaching manifold learning and dimension reduction, but can also provide specific concrete realizations. Uses Topological Analysis

Noise

an inherent sense of randomness, assumed to be unbiased and spherically symmetric

Novelty

an instance from an entirely different space or a change in current space that is suddenly occurring more often and considered the new norm

Outlier

an instance inside the probable space of known items that occurs with a very rare or low probability. generally considered noise/ edge cases

Variational Autoencoders (VAE)

are extensions of autoencoders to generate content. Variational Autoencoders map inputs to multidimensional Gaussian distributions instead of points in the latent space. Then, the decoder randomly samples a vector from this distribution to produce an output. They build general rules shaped by probability distributions to interpret inputs and to produce outputs. https://towardsdatascience.com/introduction-to-normalizing-flows-d002af262a4b

Independent probabilities

are multiplied

Attention

attention is a technique that mimics cognitive attention The effect enhances some parts of the input data while diminishing other parts — the motivation being that the network should devote more focus to the small, but important, parts of the data. Learning which part of the data is more important than another depends on the context, and this is trained by gradient descent.

Association Rule Mining, Eclat

based on lattice traversal

Association Rule Mining, FP-Growth

based on without candidate generation

Symmetric attributes

binary attributes that are equally valuably or carry the same amount of weight. negative and positive values are equally relevant

Eager learners

construct a classification model based on the given training data before receiving data for classification. It must be able to commit to a single hypothesis that covers the entire instance space. Due to the model construction, eager learners take a long time for train and less time to predict. Abstracts away from the data during training and uses this abstraction to make predictions. (decision tree induction induction and rule-based systems). Majority of computation occurs at training time.

multivariate time series forecasting problems

contain multiple variables keeping one variable as time and others will be multiple in parameters

Structured data

data conforms to a database model, which is largely characterized by the various fields that data. (retail, financial, bioinformatics, geodata)

Deep One-Class Classification (DOC)

deep-learning-based approach for one-class transfer learning in which labelled data from an unrelated task is used for feature learning in one-class classification. One-class classification trains the classifier to be able to identify out-of-class objects when given a single class sample

Classification and Regression Tree Model (CART)

defines its rule as identifying the feature in the data that best separates records into distinct classes of interest.

lazy learners

does not built models explicitly. These learners refer to any machine learning process that defers the majority of computation to consultation time. Two typical examples of lazy learning are instance-based learning and Lazy Bayesian Rules.

stemming

eliminate the suffixes. strip from back

Support Vector Data Description

finds the smallest hypersphere that contains all samples, except for some outliers .is a machine learning technique that is used for single-class classification and outlier detection. The idea of SVDD is to find a set of support vectors that defines a boundary around data. It obtains a spherically shaped boundary around a dataset and analogous to the Support Vector Classifier it can be made flexible by using other kernel functions. a data description method that can give the target data set a spherically shaped description and be used to outlier detection or classification. Multiple kinds of kernel functions (linear, gaussian, polynomial, sigmoid, laplacian)

Continuous Attributes

floating point, real, temp, height ,

GPT

generative pre-training (GPT) of a language model (2018) Generative Pre-trained Transformer 2, commonly known by its abbreviated form GPT-2, is an unsupervised transformer language model and the successor to GPT. It avoids certain issues encoding vocabulary with word tokens by using byte pair encoding. This allows to represent any string of characters by encoding both individual characters and multiple-character tokens. (2019) Generative Pre-trained[a] Transformer 3, commonly known by its abbreviated form GPT-3, is an unsupervised Transformer language model and the successor to GPT-2. It can generalize the purpose of a single input-output pair. (2020)

Lemmatization

grouping words together based on their basic dictionary definition. The base dictionary form of the word is called a lemma

Association Rule Mining, H-Confidence or all-confidence

h − confidence(X) = s(X)/max (s(i1), s(i2),..., s(ik))

Agglomerative clustering

hierarchical clustering procedure where each object starts out in a separate cluster; clusters are formed by grouping objects into bigger and bigger clusters

cosine similarity

if documents can be represented as vectors, the cosine between those vectors represents how similar they are. ignores magnitude focuses on orientation. Not proper distance cause it fails Triangle Inequality property

Asymmetric attributes

if outcomes of the states are not equally important Ex Positive for aids is more important than negative test

Graph Based Clusters (Contiguity — Based Clusters)

in cluster NN Intra-distance <<< any point outside cluster · Two objects are connected only if they are within a specified distance of each other. · Each point in a cluster is closer to at least one point in the same cluster than to any point in a different cluster. · Useful when clusters are irregular and intertwined. · This does not work efficiently when there is noise in the data, as shown in the above picture, a small bridge of points can merge two distinct clusters into one. · Clique is another type of Graph Based Cluster (Detailly explained in my future articles). · Agglomerative hierarchical clustering has close relation with Graph based clustering technique.

unstructured data

information that either does not have a predefined data model or is not organized in a predefined manner. (images, video, sensor data, web pages)

Discrete Attributes

integers, counts, binary

Data Science

involves applications of statistics, computer science, and software engineering, along with some other relevant fields (such as sociology or finance). (Top down Start with question) Statistical Analysis, Math, Scientific Method, Advanced Computing, Visualization, Domain Exp., Hacking extracting knowledge from Datasets 1. Design a Sampling scheme (Question) 2. Determine measurable quantitative attributes (Query Data) 3. Improve data Quality(Explore) 4. (Model)and Validate data 5. Communicate Result (Present)

Transformer

is a deep learning model that adopts the mechanism of self-attention, differentially weighting the significance of each part of the input data. It is used primarily in the fields of natural language processing (NLP) and computer vision (CV).

Reconstruction Models for Anomaly Detection

is a generative approach that reconstructs what it considers as normal for comparison to data that doesn't fit that definition. The difference provides a Ranking of how anomalous the new data is. suited for determining data point sets or structures A) PCA B) AutoEncoders C) prototypical clustering

Type 2 error

(False Negative) Believing that something is not real when it is. when you fail to reject the null hypothesis when you should have.

Type 1 Error

(False positive) Rejecting null hypothesis when it is true. you reject the null hypothesis when you indeed should not have.

Cosine distance

1 - cosine similarity

Cluster Types

1) Well-Separated 2) Center/prototype Based 3) Contiguous Cluster (Near-Nei) 4) Density Based 5) Conceptual

Time Series Anomalies

1. Global outliers - known as point anomalies, these outliers exist far outside the entirety of a data set 2. Contextual outliers - called conditional outliers, these anomalies have values that significantly deviate from the other data points that exist in the same context. 3. Collective outliers - When a subset of data points within a set is anomalous to the entire dataset. Individual behavior may not deviate from the normal range in a specific time series dataset. But when combined with another time series dataset, more significant anomalies become clear.

K-nearest neighbor (K-NN)

A classification method that classifies an observation based on the class of the k observations most similar or nearest to it.

Association Rule Mining, itemset

A collection of one or more items. k-itemset contains k items

Semi-structured

A database of _____ data model is a collection of nodes, each node is either a leaf or interior. structured data, but lacks a strict structure imposed by an underlying data model

Gini Coefficient

A measure of Purity in Decision Trees. Purity can be thought of as how homogenized the groupings are. 1) If we have 4 red gumballs and 0 blue gumballs, that group of 4 is 100% pure, based on color as the target. 2) If we have 2 red and 2 blue, that group is 100% impure. 3) If we have 3 red and 1 blue, that group is either 75% or 81% pure, if we use Gini or Entropy respectively.

Covariance

A measure of linear association between two variables. Positive values indicate a positive relationship; negative values indicate a negative relationship. lie between -∞ and +∞ is affected by the change in scale of the variables. So use the when the variable are on similar scales

Correlation

A measure of the extent to which two factors vary together, and thus of how well either factor predicts the other. It is dimensionless. It is a unit-free measure of the relationship between variables. between -1 and 1 it is tool for feature selection. So use the when the variable scales differ

Manhattan Distance

A method of distance measurement . = Minkowski Distance with p=1 no diagnols

Euclidean distance

A method of distance measurement using the straight line mileage between two places. = Minkowski Distance with p=2

Association Rule Mining, Generator

A set x is called a generator of y if the closure of x is y both {A} and {A, C, W} are generators of {A, C, W}.

time series data

A time series is a sequence of numbers that are ordered by a time index. This can be thought of as a list or column of ordered values. The predictions over time become less and less accurate and hence it is a more realistic approach to re-train the model with actual data as it gets available for further predictions.

Association Rule

An implication expression of the form X -> Y, where X and Y are item sets derived from transactions - Example: {Milk, Diaper} -> {Beer} is a possible rule of frequent item { Milk, Diaper, Beer} all rules have a Confidence and a Support To find one you need to: 1) Generate a Frequent Itemset 2)Generate the Rules of High confidence that binarily partition the Frequent Itemset. Note every rule of a frequent Itemset will have a high enough confidence

Anomaly

An instance outside probable space of known items

Association Rule Mining, Closed Frequent itemset

An itemset X is closed if none of its immediate supersets has the same support count as the itemset X.

Association Rule Mining, Maximal Frequent itemset

An itemset is maximal frequent if it is frequent and none of its immediate supersets is frequent. meaning all supersets are infrequent Unlike closed itemsets, maximal itemsets do not imply anything about transactions.

Association Rule Mining, Frequent itemset

An itemset whose support value is greater than or equal to a predetermined minimum support value threshold . To find every permutation is computationally crazy. In contrast Infrequent are below minimum support

Conditional/Contextual Anomaly

Anomalies specific to a context ; Space, Time, connection to a graph

One Class Support Vector Machine

One-class SVM is an unsupervised algorithm that learns a decision function for novelty detection: classifying new data as similar or different to the training set. You only have data of one class and the goal is to test new data and found out whether it is alike or not like the training data. it's a variation of the SVM that can be used in an unsupervised setting for anomaly detection. The one-class SVM finds a hyper-plane that separates the given dataset from the origin such that the hyperplane is as close to the datapoints as possible. this is different from SVM Margins

Sparsity

Only presence counts. Lack of diversity, Asymmetric

Gibbs sampling

Organize Objects into topics one object at a time wit respect to each other in the topic. a Markov chain Monte Carlo (MCMC) algorithm for obtaining a sequence of observations which are approximately from a specified multivariate probability distribution, when direct sampling is difficult. This sequence can be used to approximate the joint distribution (e.g., to generate a histogram of the distribution); to approximate the marginal distribution of one of the variables, or some subset of the variables (for example, the unknown parameters or latent variables);

Classic Anomaly Detection Algorithms

PCA Analysis One-Class SVM Support Vector Data Description K-nearest neighbor (K-NN) Kernel Density Estimation (KDE)

doc2Vec (DM)

Paragraph Vec w Dist. Mem. (DM) uses Context Words + Doc vector to predict a word in a context window

Types of anomalies

Point Anomalies Conditional/Contextual Anomalies Group/Collective Anomalies

Entropy

Probability to choosing a diverse item amongst a set. H(all c)= SUM[ - abs(c)/abs(all c) Log abs(c)/abs(all c) ] Entropy = 0 when all are the same

Hierarchical Clustering

Process of agglomerating observations into a series of nested groups based on a measure of similarity. Produces a set of nested clusters Organized as a hierarchical tree. Can be visualized as a dendrogram

Random Forest For Time Series Model

Random Forest is a popular and effective ensemble machine learning algorithm. It is widely used for classification and regression predictive modeling problems with structured (tabular) data sets Random Forest can also be used for time series forecasting, although it requires that the time series dataset be transformed into a supervised learning problem first. It also requires the use of a specialized technique for evaluating the model called walk-forward validation vs k-fold cross validation would result in optimistically biased results

Low-level features vs High Level in anomaly analysis

Refers to place in a feature hierarchy example Low - Words , Letters , sentences High - Topics and Documents

Central Limit Theorem (CLT)

Says that when n is large, the sampling distribution of the sample mean is approximately Normal

Group/Collective Anomalies

Set of collective points are abnormal

Autoencoders

Autoencoders are trained to recreate the input; in other words, the y label is the x input. One application of vanilla autoencoders is with anomaly detection. If the autoencoder can reconstruct the sequence properly, then its fundamental structure is very similar to previously seen data. On the other hand, if the network cannot recreate the input well, it does not abide by known patterns. When creating autoencoders, there a few components to take note of: The loss function is very important — it quantifies the 'reconstruction loss'. Since this is a regression problem, the loss function is typically binary cross entropy (for binary input values) or mean squared error. Determine the code size — this is the number of neurons in the first hidden layer (the layer that immediately follows the input layer). This is arguably the most important layer, because it determines immediately how much information will be passed through the rest of the layer. The performance of an autoencoder is highly dependent on the architecture. Balance representation size, the amount of information that can be passed through the hidden layers; and feature importance, ensuring that the hidden layers are compact enough such that the network needs to work to determine important features. Use different layers for different types of data. For instance, one could use one-dimensional convolutional layers to process sequences. Autoencoders are the same as neural networks, just architecturally with bottlenecks.

Reconstruction Anomaly Score

Base line error value to consider something fully matching or reconstructing a "perfect" model

K-means

Clustering is an unsupervised learning algorithm that is used for clustering

Conceptual Clusters

Clusters based on shared property or concept

Continuous Bag Of Words

Continuous bag of words creates a sliding window around current word, to predict it from "context" — the surrounding words. Each word is represented as a feature vector. After training, these vectors become the word vectors.

Density Based Clusters:

Dense regions separated by less dense regions · Cluster is a dense region of objects that is surrounded by a region of low density. · Density based clusters are employed when the clusters are irregular, intertwined and when noise and outliers are present. · Points in low density region are classified as noise and omitted. The above picture can be compared with the picture under "Graph Based clustering" for better understanding. The bridge between two circles and another small curve are eliminated. · DBSCAN is an example of Density based clustering algorithm. Density and Probabilistic models - provides a uncertainty measure or probability that tells us how much a data point is associated with a specific cluster, instead of modeling each dimension of data independently. Learn the probability distribution of large datasets. Such techniques include Generative Adversarial Networks (GANs), Variational Auto Encoders (VAEs), and Normalizing Flows. https://scikit-learn.org/stable/modules/density.html A) Histogram Estimators - aka bricks B) Kernel Density Estimation (KDE), Generative Adversarial Networks (GANs) - doesn't assume distribution has mean C) Gaussian Mixture Models (GMM), Variational Autoencoders (VAE) - assumes mean D) Marchov Chain Monte Carlo (MCMC) -estimating PDF based on probability chain. we need to determine an appropriate function for modeling posterior probability refers to the conditional probability P(A|B) of an event A given B E) Normalizing Flows

Probabilistic Latent Semantic Analysis (PLSA)

Derived from LSA there are 3 variables Documents, Words, (Topics = Hidden/Latent) P(w|d)= sum( t, P(w|t)P(t|d) ) LSA which uses only global statistics

doc2Vec (DBOW)

Dist. Bag of Words (DBOW) similar to word2vec skip grams it uses document vector to predict surrounding words as oppose to Context words to predict surrounding words in a context window

Gaussian Mixture Model (GMM)

Each Gaussian Mixture is a Cluster defined by a Gaussian function curve and each cluster has a A) A mean μ that defines its centre. B) A covariance Σ that defines its width. This woulterm-146d be equivalent to the dimensions of an ellipsoid in a multivariate scenario. C) A mixing probability π that defines how big or small the Gaussian function will be. https://towardsdatascience.com/gaussian-mixture-models-explained-6986aaf5a95

Fuzzy Clustering

Every member belongs to every member with a weight assigned 0 = not a member and 1 = absolute membership and the sum of the weights for the object must = 1

XGBoost

Extreme Gradient Boosting. decision-tree-based ensemble Machine Learning algorithm that uses a gradient boosting framework. In prediction problems where it comes to small-to-medium structured/tabular data,using ensemble tree methods and gradient descent to apply the principle of boosting weak learners are considered best-in-class right now. Parallel processing+ Tree Pruning + handle Missing values + regularization

Amazon Blazing text

FastText with GPU enhancements 1) CBOW and Skip Grams generate Embeddings (Unsupervised) 2) Given Vocabulary for text Classification (Supervised)

Association Rule Mining, Support Count

Frequency of an itemset occurrence including subsets

generative models

Generative models can generate new data instances. Generative models capture the joint probability p(X, Y), or just p(X) if there are no labels. They contrast with discriminative models. A generative model could generate new photos of animals that look like real animals, while a discriminative model could tell a dog from a cat. unsupervised models that summarize the distribution of input variables may be able to be used to create or generate new examples in the input distribution generative models include Latent Dirichlet Allocation, or LDA, and the Gaussian Mixture Model, or GMM.Restricted Boltzmann Machine, or RBM, and the Deep Belief Network, or DBN. Variational Autoencoder, or VAE, and the Generative Adversarial Network, or GAN.

Latent Dirichlet Allocation (LDA)

Generative statistical Model that will explain why some parts are similar using unobserved group = Topic to explain a (set of observations) = words in a document. In This Topic every word gravitates to a Topic and the collection of words gravitate to specific topics Needs: a) Set of Topics b) Stop-Words (Normalize) c) Stemming (Normalize) d) Lemmatization (Normalize) but it's very hard to tune, and results are hard to evaluate.

Glove Embeddings

Global Vectors -The advantage of GloVe is that, unlike Word2vec, GloVe does not rely just on local statistics (local fixed/surrounding context information of words), but incorporates global statistics (word co-occurrence over given corpus ) to obtain word vectors. "You can derive semantic relationships between words from the co-occurrence matrix." GloVe captures both global statistics and local statistics of a corpus, in order to come up with word vectors aka h a principled loss function which uses both

Connectivity-based Clustering

Graph Based or also known as Hierarchical Clustering when clusters are nested can be generated using two types of methods 1) Top-Down Divisive Approach 2) Bottom-Up Agglomerative approach

Association Rules

IF <antecedent> THEN <consequent> Detect unexpected Relations, Medical Informatics, Market Analysis There are three popular algorithms of Association Rule Mining, Apriori (based on candidate generation), FP-Growth (based on without candidate generation) and Eclat (based on lattice traversal).

Association Rule Mining, Aprior

If an itemset is frequent, then all of its subsets must also be frequent. (Anti-Monotone property of Support) = ∀W,V : (V ⊆W)⇒s(V) ≥ s(W) based on candidate generation. Can be inefficient, especially for dense datasets. Apriori is a program to find association rules and frequent item sets (also closed and maximal as well as generators) with the Apriori algorithm [Agrawal and Srikant 1994], which carries out a breadth first search on the subset lattice and determines the support of item sets by subset tests. This implementation is pretty fast as it uses a prefix tree to organize the counters for the item sets. However, Apriori is outperformed on basically all data sets by depth-first algorithms like Eclat or FP-growth.

Well-Separated Clusters

Intra distance <<< inter distance · Distance between any two points in different groups is larger than the distance between any two points in the same group. · These clusters need not be globular but, can have any shape. · Sometimes a threshold is used to specify that all the objects in a cluster must sufficiently close to one another. Definition of a cluster is satisfied only when the data contains natural clusters. Could be considered Distribution based

t-Distributed Stochastic Neighbor Embedding (t-SNE)

Is a statistical method for visualizing high-dimensional data by giving each datapoint a location in a two or three-dimensional map.

Association Rule Mining, Frequent Itemset Generator

Is the Graph formed out of all item sets possibilities minsup =0 means every subset is frequent 2^d-1 where d = max length of largest set. if minsup huge the number of rules are small

semi-supervised learning

Machine learning using a combination of supervised and unsupervised learning techniques. A learning problem that involves a small number of labeled examples and a large number of unlabeled examples.

ARIMA Model

Model used in Time Series Forecasting. It is Statistical Based. It stands for AutoRegressive Integrated Moving Average.

Deep Anomaly Detection Algorithms

Deep Autoencoder Variants Deep One-Class Classification (DOC) Generative Adversarial Networks (GANs)

Anomaly Score

score is created using an anomaly/id and the new instance (input_data) for which you wish to create an anomaly score. When you create a new anomaly score, BigML.io will automatically compute a score between 0 and 1. The closer the score is to 1, the more anomalous the instance being scored is.

automated anomaly detection systems

should include detection, ranking, and grouping of data, eliminating the need for large teams of analysts.

BERT

Bidirectional Encoder Representations from Transformers (BERT) is a transformer-based machine learning technique for natural language processing (NLP) pre-training developed by Google.

Center/Prototype Based clusters

Center 1-Xc1 <<< Centerc-Xc1 distance · If the data is numerical, the prototype of the cluster is often a centroid i.e., the average of all the points in the cluster. · If the data has categorical attributes, the prototype of the cluster is often a medoid i.e., the most representative point of the cluster. · Objects in the cluster are closer to the prototype of the cluster than to the prototype of any other cluster. · These clusters tend to be globular. · K-Means and K-Medoids are the examples of Prototype Based Clustering algorithms

Manifold assumption

Data lives on a low dimensional level that may be a non linear non convex Manifold. So you could match similarity to the shape of the Manifold

Normalizing Flows

normalizing flows is a series of simple functions which are invertible, or the analytical inverse of the function can be calculated. For example, f(x) = x + 2 is a reversible function because for each input, a unique output exists and vice-versa whereas f(x) = x² is not a reversible function. Such functions are also known as bijective functions. The normalizing flows transform a complex data point such as MNIST Image to a simple Gaussian Distribution or vice-versa. Normalizing flows offers various advantages over GANs and VAEs. Some of them are listed as follows:- A) The normalizing flow models do not need to put noise on the output and thus can have much more powerful local variance models. B) The training process of a flow-based model is very stable compared to GAN training of GANs, which requires careful tuning of hyperparameters of both generators and discriminators. C) Normalizing flows are much easier to converge when compared to GANs and VAEs Disadvantages of Normalizing Flows:- While flow-based models come with their advantages, they also have some shortcomings as follows:- A) Due to the lackluster performance of flow models on tasks such as density estimation, it is regarded that they are not as expressive as other approaches. B) One of the two things required for flow models to be bijective is the volume preservation over transformations, which often leads to very high dimensional latent space, which is usually harder to interpret. C) The samples that are generated through flow-based models are not as good when compared to GANs and VAEs..

Point Anomaly

one data Point is not normal

univariate time series forecasting problems

only two variables in which one is time and the other is the field to forecast

population proportion

p = Ratio of members of a population with a particular characteristic to the total members = n of the population To accurately assess 1) size of sample has to be less than 5% of population 2) satisfying np(1-p) >=10 approximates that a normal distribution exists so we can compare it to Z Score dist

Association Rule Mining, Cross Support

r(X) = min (s(i1), s(i2),..., s(ik)) /max (s(i1), s(i2),..., s(ik))