Probability

What are disjoint events?

A case when any two particular events cannot occur at the same time

What are combinations?

A combination is a selection of items from a collection, such that (unlike permutations) the order of selection does not matter. If the set has n elements, the number of k-combinations is equal to nCr = n! / [r!(n - r)!]

What are permutations?

A permutation of a set is, loosely speaking, an arrangement of its members into a sequence or linear order. When writing permutations, we use the notation nPr, where n represents the number of items to choose from, P stands for permutation and r stands for how many items you are choosing. To calculate the permutation using this formula, you would use nPr = n! / (n - r)!

What are random variables?

A random variable is understood as a measurable function defined on a probability space that maps from the sample space to the real numbers.

What are some rules associated with the expectation operator? Think expectation of sum functions and product functions.

For any continuous or discrete random variable and constant a and linear function g, E (a g (X)) = aE (g (X)) For any constants a, b and any continuous or discrete random variable and linear functions g1, g2, where independence of X and Y does NOT matter, E (a g1 (X) + b g2 ( Y )) = aE (g1 (X)) + bE (g2 (Y )) If two random variables are independent, then the expectation of the product factors into a product of expectations. E (g (X) h (Y )) = E (g (X)) E (h (Y ))

What is the mean and variance for the Uniform random variable?

Given lower bound a and upper bound b, the mean value is (a+b)/2 and the variance is (b-a)^2 / 12

What is the mean and variance for the Binomial random variable?

Given n is the number of trials, and p is the probability of success, the mean number of trials with success out of n trials is np, and the variance is np(1-p).

What is the mean and variance for the Geometric random variable?

Given p is the probability of success, the mean number of trials before success is achieved is 1/p and the variance is (1-p)/p^2

What is the mean and variance for the Gaussian random variable?

Given the mean and standard deviation, the mean value is the mean and the variance is the standard deviation.

What is the mean and variance for the Poisson random variable?

Given λ is the average rate of some outcome in some time frame, the mean outcome in some time frame is λ and the variance is λ.

What is the mean and variance for the Exponential random variable?

Given λ is the average rate of some outcome in some time frame, the mean time between each outcome is 1/λ and the variance is 1/λ^2

What is the probability of independent events (A and B, A or B)?

If the occurrence of event A doesn't affect the occurrence of event B, these events are called independent events. P (A ꓵ B) = P (A) * P (B) P(A ꓴ B) = P(A) + P(B) - P(A ꓵ B)

Explain a geometric random variable.

Imagine that we take a biased coin and flip it until we obtain heads. If the probability of obtaining heads is p and the flips are independent then the probability of having to flip k times is: P(k flips) = (1 − p)^(k−1) * p This reasoning can be applied to any situation in which a random experiment with a fixed probability p is repeated until a particular outcome occurs, as long as the independence assumption is met. In such cases the number of repetitions is modeled as a geometric random variable.

Give common examples of continuous random variables.

A uniform random variable models an experiment in which every outcome within a continuous interval is equally likely. As a result the pdf is constant over the interval. The pdf of a uniform random variable with domain [a, b] is 1/(b - a) for x between [a, b] and 0 otherwise. Exponential random variables are often used to model the time that passes until a certain event occurs. Examples include decaying radioactive particles, telephone calls, earthquakes and many others. The pdf is λe^(−λx) if x is >= 0, and 0 otherwise. The Gaussian or normal random variable is arguably the most popular random variable in all of probability and statistics. It is often used to model variables with unknown distributions in the natural sciences. This is motivated by the fact that sums of independent random variables often converge to Gaussian distributions. This phenomenon is captured by the Central Limit Theorem. The pdf of a Gaussian is parameterized by mean and standard deviation. An annoying feature of the Gaussian random variable is that its cdf does not have a closed form solution, in contrast to the uniform and exponential random variables. This complicates the task of determining the probability that a Gaussian random variable is in a certain interval. One must express the probability in terms of the cdf a standard Gaussian (mean is zero, stdev is one) to solve the problem. Recall the integral of an entire pdf is equal to one.

What is conditional probability?

Conditional probability is a measure of the probability of an event occurring given that another event has (by assumption, presumption, assertion or evidence) occurred. If the event of interest is A and the event B is known or assumed to have occurred, "the conditional probability of A given B", or "the probability of A under the condition B", is usually written as P(A | B). P(A | B) may or may not be equal to P(A) (the unconditional probability of A). If P(A | B) = P(A), then events A and B are said to be "independent": in such a case, knowledge about either event does not give information on the other. P(A | B) typically differs from P(B | A). The common definition is the Kolmogorov definition, which states given two events A and B with the unconditional probability of B being greater than zero, P(B) > 0, the conditional probability of A given B is defined as the quotient of the probability of the intersection of events A and B, and the probability of B. P(A|B) = P(A∩B) / P(B) There is also conditioning on a random variable. Let X be a random variable; we assume for the sake of presentation that X is finite, that is, X takes on only finitely many values x. Let A be an event. The conditional probability of A given X is defined as the random variable, written P(A|X), that takes on the value P(A | X = x) whenever X = x

What are discrete random variables? Give common examples of discrete random variables.

Discrete random variables take values on a finite or countably infinite subset of R such as the integers. They are used to model discrete numerical quantities: the outcome of the roll of a die, the score in a basketball game, etc. To specify a discrete random variable it is enough to determine the probability of each value that it can take, known as a probability mass function. In contrast to the case of continuous random variables, this is possible because these values are countable by definition. The most well known discrete random variables are Bernoulli, Geometric, Binomial, and Poisson. Bernoulli random variables are used to model single trial experiments that have two possible outcomes. Geometric random variables model experiments in which a fixed probability p is repeated until a particular outcome occurs. A binomial random variable are used to model the number of positive outcomes of n trials modeled as independent Bernoulli random variables. The Poisson random variable expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant mean rate and independently of the time since the last event.

What is conditional expectation?

In probability theory, the conditional expectation of a random variable is its expected value - the value it would take "on average" over an arbitrarily large number of occurrences - given that a certain set of "conditions" is known to occur. Depending on the nature of the conditioning, the conditional expectation can be either a random variable itself or a fixed value. With two random variables, if the expectation of a random variable X is expressed conditional on another random variable Y without a particular value of Y being specified, then the expectation of X conditional on Y, denoted E[X | Y], is a function of the random variable Y and hence is itself a random variable. Alternatively, if the expectation of X is expressed conditional on the occurrence of a particular value of Y, denoted y, then the conditional expectation E[X | Y=y] is a fixed value.

How is conditional probability used in statistical inference?

In statistical inference, the conditional probability is an update of the probability of an event based on new information. The wording "evidence" or "information" is generally used in the Bayesian interpretation of probability. The conditioning event is interpreted as evidence for the conditioned event. That is, P(A) is the probability of A before accounting for evidence E, and P(A|E) is the probability of A after having accounted for evidence E or after having updated P(A). This is consistent with the frequentist interpretation.

How do you sample from a continuous random variable distribution?

Let X be a continuous random variable with cdf FX and U a random variable that is uniformly distributed in [0, 1] and independent of X. Recall FX(x) is the probability that X will take on a value less than or equal to x. To make it possible to sample from an arbitrary distribution with a known cdf by applying a deterministic transformation to uniform samples, you use a technique called inverse-transform sampling. You compute the inverse of the cdf, FX^-1, and for every sample u randomly generated from U, set x = FX^-1(u). If FX is not invertible for certain intervals, redefine FX^-1 as the min{FX(x) = u}. It has been mathematically proven with these definitions that the distribution of FX^-1(U) is the same as the distribution of X.

What is the expectation of continuous random variables?

Let X be a continuous random variable. The expected value of a function g (X) with probability density function fX is E (g (X)) := integral[g (x) fX (x) dx] for x=-∞ to x=∞

How do you sample from a discrete random variable distribution?

Let X be a discrete random variable with pmf pX and U a uniform random variable with unit interval [0, 1]. Our aim is to transform a sample from U so that it is distributed according to pX. We denote the values that have nonzero probability under pX by x1, x2, etc. We can treat the sample space of U as probabilities (interval is between 0 and 1). So for each xi in X, we specify an interval within [0, 1] as pX(xi-1) to pX(xi), and every u randomly sampled from U that is in this interval is mapped to xi. That way the probability of assigning a random sample from U to xi is pX(xi). Recall the sum of pX(xi) for all xi in discrete random variable X is one. So after mapping all randomly generated samples from U to an xi according to pX, we are able to sample from discrete random variable X according to its probability mass function!

What is the expectation of discrete random variables?

Let X be a discrete random variable with range R. The expected value of a function g (X) with probability mass function pX is E (g (X)) := sum[ g (x) pX (x) ] for all x in X

What is Markov's inequality? What is Chebyshev's inequality?

Markov inequality quantifies the intuitive idea that if a random variable is nonnegative and small then the probability that it takes large values must be small. Let X be a nonnegative random variable. For any positive constant a > 0, P (X ≥ a) ≤ E (X) / a This should only be used if the standard deviation of the random variable is unknown. If it is known, use Chebyshev's inequality. P (|X − E (X)| ≥ a) ≤ Var (X) / a^2

What are continuous random variables? How do probability density functions relate to them?

Physical quantities are often best described as continuous: temperature, duration, speed, weight, etc. In order to model such quantities probabilistically we could discretize their domain and represent them as discrete random variables. However, we may not want our conclusions to depend on how we choose the discretization grid. We cannot characterize the probabilistic behavior of a continuous random variable by just setting values for the probability of X being equal to individual outcomes, as we do for discrete random variables. Instead, we consider events that are composed of unions of intervals of R. We model continuous random variables with cumulative distribution functions, which defines a function that determines the probability of random variable X taking on a value less than or equal to x. FX (x) := P (X ≤ x) We can then use unions of intervals like so: P (a < X ≤ b) = P (X ≤ b) − P (X ≤ a) = FX (b) − FX (a). Now, to find the probability of X belonging to any particular set, we only need to decompose it into disjoint intervals. The probability of individual points are zero. If the cdf of a continuous random variable is differentiable, its derivative can be interpreted as a probability density function. This density can then be integrated to obtain the probability of the random variable belonging to an interval or a union of intervals

What is Bayes theorem?

The Bayes theorem describes the probability of an event based on the prior knowledge of the conditions that might be related to the event. If we know the conditional probability, we can use the Bayes rule to find out the reverse probabilities. P(A | B) = [P(B | A)*P(A)] / P(B) where P(A | B) is the posterior probability

What is the covariance of two random variables? Given the covariance, what is the variance of the sum of two random variables? What is the Pearson correlation coefficient?

The covariance of two random variables describes their joint behavior. It is the expected value of the product between the difference of the random variables and their respective means. Intuitively, it measures to what extent the random variables fluctuate together. Cov(X, Y) := E ((X − E (X)) (Y − E (Y ))) = E (XY) − E(X) E (Y ) If Cov (X, Y ) = 0, X and Y are uncorrelated. Var (X + Y ) = Var (X) + Var (Y ) + 2 Cov (X, Y ) NOTE: Uncorrelation does not imply independence! The covariance does not take into account the magnitude of the variances of the random variables involved. The Pearson correlation coefficient is obtained by normalizing the covariance using the standard deviations of both variable. ρ(X,Y) := Cov (X, Y ) / (σX*σY) A useful interpretation of the correlation coefficient is that it quantifies to what extent X and Y are linearly related. In fact, if it is equal to 1 or -1 then one of the variables is a linear function of the other!

What is the expectation of a function g(X, Y) conditioned on X = x for both cases of Y being discrete and continuous.

The expectation of g conditioned on the event X = x for any fixed value x can be computed using the conditional pmf or pdf of Y given X. If Y is discrete, E (g (X, Y )|X = x) = sum[ g (x, y) * p (y|x) ] for y in Y If Y is continuous, E (g (X, Y )|X = x) = integral[ g (x, y) f(y|x) dy ] from y=-∞ to y=∞ where f is the pdf of g

What is the expectation operator?

The expectation operator allows us to define the mean, variance and covariance rigorously. It maps a function of a random variable or of several random variables to an average weighted by the corresponding pmf or pdf. It is a generalization of the weighted average and intuitively is the arithmetic mean of a large number of independent realizations of that variable. The expected value of random variable X is the probability weighted sum of all possible outcomes.

What is the first moment of a random variable X?

The first moment is the mean, and for a random variable X the mean is the expected value of X: E[X] If the distribution of a random variable is very heavy tailed, which means that the probability of the random variable taking large values decays slowly, its mean may be infinite. The mean is often interpreted as representing a typical value taken by the random variable. However, the probability of a random variable being equal to its mean may be zero! For instance, a Bernoulli random variable cannot equal 0.5.

How do you determine statistical independence using conditional probability?

The following statements demonstrate statistical independence between events A and B: P(A∩B) = P(A)P(B) P(A | B) = P(A) where P(B) is not 0 P(B | A) = P(B) where P(A) is not 0

What are some descriptive statistical quantities for random variables?

The mean is the value around which the distribution of a random variable is centered. The variance quantifies the extent to which a random variable fluctuates around the mean. The covariance of two random variables indicates whether they tend to deviate from their means in a similar way.

What is the mean and variance for the Bernoulli random variable?

The mean outcome from a Bernoulli trial is p, the probability of success, and the variance is p(1 - p).

What is the second moment of a random variable X? What is the second centered moment of a random variable X's distribution?

The mean square or second moment of a random variable X is the expected value of X^2 : E X^2. The mean square of the difference between the random variable and its mean is called the variance of the random value (the variance is itself a mean, the mean deviation from the mean). It quantifies the variation of the random variable around its mean and is also referred to as the second centered moment of the distribution. The square root of this quantity is the standard deviation of the random variable. Var (X) := E [ (X − E (X))^2 ] = E( X^2 ) − [E (X)]^2 The variance of linear functions with constants a, b is: Var(aX + b) = a^2 * Var(X)

What is the intersection of events?

The probability of both event A and event B occurring jointly, P (A ꓵ B).

What is the union of events?

The probability of event A or event B occurring, P (A ꓴ B)

What is the sample space?

The sample space is nothing but the collection of all possible outcomes of an experiment. This means that if we perform a particular task again and again, all the possible results of the task are listed in the sample space.