Unit 2: Auto Regressive and Moving Average (ARMA) Model

property of lag 0 ACF for white noise

This aligns with the property of the auto-covariance function of a white noise, which is nonzero at lag 0 and it's equal to the variance, but is 0 otherwise.

MLE formula to minimize

To estimate the AR and MA coefficients, we minimize this likelihood function, more precisely, the log of the likelihood function.

why do we prefer that processes are invertible?

We prefer the invertible ARMA processes in practice because if we can invert an MA process to an AR process, we can find the value of Zt which is not observable, based on all past values of Xt which are observable. If a process is non-invertible, then in order to find the value of Zt, we have to know all future values of Xt.

can Yule Walker be used to estimate order of AR(p)?

YES

For ARMA parameter estimation, do you de-mean? Are p and q fixed?

Yes and Yes

Can MA be used on ARMA models?

Yes, if causal

invertible process

Zt is white noise process

stochastic

a random process - an expected value + variation

ARMA stands for

auto-regressive moving average

why do we perform a histogram/Q-Q plot with ARMA/ARIMA?

b/c we assume normality

what is a causal process?

can invert MA to AR and AR to MA model

What's a downside to AICC?

computationally very expense

stationary solution to ARMA equation if this condition met

for all values | z | = 1, phi(z) cannot = 0

How can we use the Yule Walker equations?

for finding the phi coefficients of the AR model

forecasting ARIMA model

forecasting in ARIMA <> forecasting in ARMA b/c ARMA processes assumed causal, and can't do that w/ ARIMA

Sample PACF

good for identifying a 'good' model

what are the conditions of an ARMA process?

if stationary for every t

What conditions make a time series ideal for Yule Walker, asides from the requirements of white noise w/ mean zero and constant variance?

large length of time series or large sample size

as phi increases, is it more or less likely that a process approaches causality?

less likely See example - top time series is stationary while bottom is NOT

What does the last equation here mean?

means that for h > p, the variance ~ 1/p AN = asymptotic normal distribution

what does side = 1 mean?

means we're generating an MA process

what method do you use for an AR process?

method = recursive

are all stationary processes causal?

no

another approach for evaluating a prediction is

seeing if the observed values fall within the prediction bands

how do you find the psi coefficients?

solve a system of linear equations

what part of the notation refers to the autoregressive part?

the B term

what happens to the co-variance between two points in a stationary process as the lag approaches infinity?

the co-variance approaches 0

What's the d in the ARIMA model?

the d represents how many differences - differences can remove trend

prediction accuracy measures

the y* is the predicted values The capital y_hat is mean fit response MSPE is appropriate for validation prediction accuracy for model using the best linear prediction approach. But it depends on scales, and it's sensitive to outliers. MAE is not appropriate to evaluate prediction accuracy of the best linear prediction, and it depends on skills but it is robust to outliers. MAPE is not appropriate again, to evaluate prediction accuracy of the best linear prediction, but it does not depend on the scale, and it is robust to outliers. PM or the percentage measure, it's a appropriate for evaluating prediction accuracy of the best linear prediction and it does not depend on the scale. The PM measure is reminiscent of the regression R squared used in the linear regression. It can be interpreted as the proportion between the variability in the prediction and the variability in the new data. While MAE and MAPE are commonly used to evaluate prediction accuracy, I recommend using the precision measure.

ARMA order and ACF below confidence bands

we do not expect the sample autocorrelation to be approximately 0 for lags larger than the order of the MA process, but to be approximately 0. *only for stationary processes

What do we use to estimate MA parameters, the theta parameters?

we use the Innovations algorithm

ARMA autocovariance function

where psi = coefficients to linear process

Can ARIMA be used on non-stationary processes?

yes

are all causal processes stationary?

yes

is there a significant difference b/w predicting 10 days ahead, and 10 days ahead (But one day at a time)?

yes

if auto-regressing, will you have to filter your data?

yes - since this ARMA model is order 2, you must filter p:end in your indices where p is order

Can Yule Walker determine confidence intervals and statistical significance of parameters?

yes!

identifying order

AR(p) from PACF MA(q) from ACF For an ARMA process, it can be shown that the PACF has the property, that it's the 0 for lags larger than the order p of the AR process. Thus, we can identify the lack of the AR process using the PACF plot. To summarize, if we have an AR process, we can identify p using PACF. And if we have an MA process, we can identify q using ACF. Unfortunately, there are no such simple rules for ARMA(p,q) processes in general.

ARIMA limitations

ARIMA modeling is easy to implement, but it captures the non-stationary in trend assuming similarity among prior observations and thus it can mispredict if there are large changes in the trend. On the other hand, the trend plus ARIMA estimation approach is more difficult to implement, particularly if interested in obtaining confidence bands also, but it can capture long-memory trends. Prediction using ARIMA can perform well if only short time prediction periods are considered.

PACF and confounding

Another important measure of the dependence for a time series that accounts for disconfounding is the partial autocorrelation function.

In the ED data example, what methods do we use to test for correlation?

Box-Pierce and Ljung-Box

AR coefficient estimation via linear regression

Constraint - must assume normality

Confounding

Correlation between two observations in a time series at two time points can result from a mutual linear dependence on observations at other time points called confounding.

what's a less computationally expensive order selection method than AICC?

Extended auto correlation function

coefficient correlation example

From this examples, and other examples, we see that although Xt is modeled as a moving average model, thus, a linear combination of white noise. Then Xt and Xt-1 are still dependent on the sample sequence of white noise. Thus, they are correlated. This correlation is reflected in the ACF plots.

PACF above bands and order

If we were to use the PACF plot to identify the lag of the AR process, we identity an AR process with order one. However, the property of the PACF for an AR process does not hold for non-stationary processes. See non-starionary and stationary, both AR(2) plots The non-stationary makes it look like order 1 when it's order 2

Are the expectation and covariance of the ARIMA process uniquely determined?

Importantly, the expectation and the covariance of the ARIMA process are not uniquely determined.

How does ARIMA differ from ARMA?

In ARMA d = 0 while in ARIMA d > 0 Also, a causal ARMA process will not be causal as ARIMA b/c there are d roots on the unit circle

statistical properties of MLE

Last, we can use the property of the MLEs, the estimated maximum likelihood estimators, particular the property of asymptotically normally distributed, regardless of the assumption on the noise Zt. The MLEs are asymptotically unbiased, that is, for large sample size, the expectation of the MLEs is close to the true parameter values. The covariance matrix of the estimator is directly dependent on the covariance of the time series. This asymptotic distribution can be used for statistical inference on the AR and MA coefficients, as well as for the asymptotic distribution of the ACF and PACF.

Consequence of ACF estimate versus exact value

Moreover, its estimate is not exact, but it's a random variable with a distribution as I will explain in more detail in a next lesson. Because of this, we do not expect the sample autocorrelation to be exactly 0 for lags larger than the order of the MA process, but to be approximately 0. Thus, the ACF plot indicates that the order of the MA process is 1, which is the order of the process we actually simulated.

consider the sample model

Mt is trend St is seasonality Zt is white noise In practice: Because there are many orders to determine those of ARIMA P, D, Q, along with those of a seasonal ARIMA, capital P, capital D, capital Q, such a model is difficult to implement.

To apply ARMA, do the random variables need a normal distribution?

NO Note, that the distribution of the random variables does not need to be normal.

Can Yule Walker be used for MA and ARMA?

NO!

Can the ACF and PACF plots predict p and q for an ARMA process?

No The PACF plot here suggests AR(p) = 3 and the ACF plot here suggests MA(q) = 4. However, it's (2, 2)

would you expect to see periodic patterns in PACF?

No - by design should exclude

If you fit a time series with AR alone and then ARMA, will both models have the same AR(p) that minimizes AIC?

No; in the ED data example the AR alone minimized AIC with p order 6 but with ARMA the p order was 5.

ARMA notation

Phi is a polynomial of order p with coefficients given by the AR portion of the model. Theta is a polynomial of order q with coefficients given by the MA portion

AR linear regression process

1) calculate actuals PACF and see if time series order aligns with order chosen for model 2) calculate coefficients w/ right order 3) see if time series - residuals is inside confidence bands and shows AR(0)

process used in IBM example

1) check log(time series) with ACF/PACF 2) check log(time series difference order = 1) with ACF/PACF (lags should give guidance to AR order needed) 4) check ARMA with ACF/PACF/Histogram/QQ/Ljung and Box-Pierce

process to estimate ARMA parameters

1. The AR coefficients denoted with phi1 to phip and the MA coefficients denoted with theta1 to thetaq are unknown parameters. But if the process has non-zero mean, then, mu, the mean of the process, is a parameter, along with a variance of the y noise Zt. 2. Commonly, we estimate them in parameter mu first, and 3. subtract mu from the process, assuming a zero mean process. 4.Thus, we only estimate the AR and MA coefficients, and the variance parameter.