Reading 7: Introduction to Linear Regression
Mean Square Regression (MSR)
The sum of squares regression divided by the number of independent variables k; in a simple linear regression, k = 1 and MSR = SSR [LOS 7.e]
SSR (sum of squared regression)
The sum of the squared difference bewteen (1) the value of the dependent variable based on the estimated regression line and (2) the mean of the dependent variable. ---> Explained variations [LOS 7.d]
SSE (sum of squared errors)
The sum of the squared vertical distances between the estimated and actual Y-values --> Unexplained variations Also called Sum of squared Residuals [LOS 7.d]
What can be used to test if slope coefficient is statistically meaningful?
To test the hypothesis about a slope, a t-distributed test statistic can be used: b^1=the estimated value of the slope coefficient b1=the hypothesized value of the slope coefficient sb^1 = the standard error of the slope coefficient The test statistic is t-distributed with n−k−1 degrees of freedom. In a simple linear regression, k=1, and thus the degrees of freedom is n−2. [LOS 7.f]
In a regression with loan rate as the dependent variable and debt-to-income ratio as the independent variable, the regression coefficient for the independent variable is 0.7775. In addition, the standard deviation of the independent variable is 0.0454 and the standard error of the estimate is 0.0255. There are 10 observations. 1. Calculate the standard error of the slope coefficient. 2. Test whether the slope coefficient of regression is statistically significant at the 0.05 level of significance.
1. First, calculate the variation of the independent variable using the standard deviation of the independent variable: 0.0454=√[∑(Xi−X¯)^2/(n−1)] √∑(Xi−X¯)^2 = 0.0454x√9 = 0.1362 Standard errir if the slope coefficient is: sb^1 = SEE/0.1362 = 0.0255/0.1362 = 0.1872 2. H0:b1=0vs.Ha:b1≠0 t=(b^1−b1)/sb^1 with 10−2=8 degrees of freedom. α=0.05 The critical t value is 2.306 (from the t-distribution with df=8) Reject the null hypothesis if the calculated t statistic is greater than 2.306. t=0.7775−00.1872=4.15 Since the calculated t-statistic exceeds the critical t value, reject the null hypothesis. The slope coefficient of regression is statistically different from 0 at the 0.05 level of significance. [LOS 7.f]
4 assumptions of simple linear regression
1. Linearity: The relationship between the dependent variable and the independent variable is linear. 2. Homoscedasticity: The variance of the residuals is constant for all observations. 3. Independence: The pairs (X,Y) are independent of each other. This implies the residuals are uncorrelated across observations. 4. Normality: The residuals are normally distributed. [LOS 7.c]
All else equal, an increase in which of the following will most likely result in a lower F-statistic? A. Residual sum of squares B. Number of observations C. Mean regression sum of squares
A. For a linear regression with one independent variable, the F-statistic is calculated as follows: F=Mean regression sum of squares/Mean squared error=RSS/1/SSE/(n−2) where n is the number of observations RSS is the regression sum of squares SSE is the sum of squared errors, or residual sum of squares All else equal, increases in the value of RSS or n will produce a higher F-statistic. By contrast, a higher SSE will result in a lower F-statistic. [LOS 7.e]
Consider the following statement: In a simple linear regression, the appropriate degrees of freedom for the critical t-value used to calculate a confidence interval around both a parameter estimate and a predicted Y-value is the same as the number of observations minus two. The statement is: A. justified. B. not justified, because the appropriate of degrees of freedom used to calculate a confidence interval around a parameter estimate is the number of observations. C. not justified, because the appropriate of degrees of freedom used to calculate a confidence interval around a predicted Y-value is the number of observations.
A. In simple linear regression, the appropriate degrees of freedom for both confidence intervals is the number of observations in the sample (n) minus two. (LOS 7.d)
Standard error of the estimate and how it affects the model's fit
An absolute measure of the distance of the observed dependent variable from the regression line. The lower the SEE, the better the model fit. [LOS 7.e]
A violation of the independence assumption will lead to...
Estimations of variance, as well as estimates of the model parameters, will not be correct. [LOS 7.c]
What is used to test if model is statistically meaningful?
F-Test: F= MSR/MSE with df of regression df1=k = 1 (numerator) and df error df2=n-k-1 = n-2 (denominator) F = SSR/SSE/(n-2) = SSRx(n-2)/SSE [LOS 7.e]
Covariance Formula
Product of difference in each observation from mean of dependent (Y) and independent (X) variable, divided by sample size (n for population covariance, n-1 for sample covariance).
Coefficient of Determination
R-Squared: The percentage of the variation of the dependent variable that is explained by the independent variable. R2 = SSR/SST [LOS 7.e]
If y represents the dependent variable and x the independent variable, this relationship is described as the regression of ... on ....
Regression of y on x. [LOS 7.a]
Slope interpretation of lin-log model
The absolute change in the dependent variable for a relative change in the independent variable. A 1 percent increase in x is expected to lead to a (b1/100) unit increase in y. [LOS 7.h]
The equation for the confidence interval for a predicted value of Y
The challenge with computing a confidence interval for a predicted value is calculating sf. On the Level I exam it's highly unlikely that you will have to calculate the standard error of the forecast (it will probably be provided if you need to compute a confidence interval for the dependent variable). [LOS 7.g]
What is another implication of the linearity assumption?
The independent variable must not be random (i.e., it is non-stochastic). This is because the linear relationship between the dependent variable and the independent variable would not exist if the independent variable is random. Consequently, the residuals are random. When the dependent variable is plotted against the independent variable, the residuals should not exhibit a pattern. [LOS 7.c]
A violation of the linearity assumption will lead to...
The model will underestimate or overestimate the dependent variable at certain points, and thus produce biased results. [LOS 7.c]
Given ABC regression equation: ABC^ = -2.3% + 0.64(S&P500^) Calculate a 95% prediction interval on the predicted value of ABC excess returns with n=36. Suppose the standard error of the forecast is 3.67, and the forecast value of S&P 500 excess returns is 10%.
The predicted value for ABC excess returns is: ABC^ = -2.3% + 0.64x10% = 4.1% The 5% two-tailed critical t-value with 34 degrees of freedom is 2.03. The prediction interval at the 95% confidence level is: ABC^-2.03x3.67<ABC<ABC+2.03x3.67 =4.1%-7.4501<ABC<4.1%+7.45 or -3.4% to 11.6% We can interpret this to mean that, given a forecast value for S&P 500 excess returns of 10%, we can be 95% confident that the ABC excess returns will be between -3.4% and 11.6%. [LOS 7.g]
Slope interpretation of log-log model
The relative change in the dependent variable for a relative change in the independent variable. A 1% increas in x is expected to lead to a 1% increase in y. [LOS 7.h]
Slope interpretation of log-lin model
The relative change in the dependent variable for an absolute change in the independent variable. A 1 unit increase in x is expected to lead to a (100xb1)% increase in y. [LOS 7.h]
If there is only one independent variable in the regression, what is the coefficient of determination equal to?
The square of the correlation between the dependent variable and the independent variable: R2=-sqrt(r2) [LOS 7.e]
Covariance and Correlation
[LOS 7.b]
Linear Intercept
[LOS 7.b]
Slope coefficient formula
[LOS 7.b]
What is the degrees of freedom of the regression equal to?
df = k = number of slope parameters in the regression [LOS 7.e]
Standard error of the intercept
supplement to [LOS 7.f]
Standard error of the forecast
supplement to [LOS 7.g]
The following data are given: SSR = 0.011197, SSE = 0.005193, SST = 0.016390 MSR = 0.011197, MSE = 0.000649 df1 = 1, df2 = 8 1. Calculate the coefficient of determination. 2. Calculate the standard error of the estimate. 3. At a 5% level of significance, do we reject the null hypothesis that the slope coefficient is equal to zero? 4. Based on the answers to Parts 1 to 3, evaluate this simple regression model.
1. R2=SSR/SST=0.011197/0.016390=0.683 2. se=√MSE=√0.000649=0.0255 3. H0:b1=0 vs. Ha:b1≠0 F=MSR/MSE=SSR/SSE/(n−2) with 1 and 8 degrees of freedom. α=0.05 The critical F value is 5.32 (from the F-Distribution with df1=1, df2=8). Reject the null hypothesis if the calculated F statistic is greater than 5.32. F=MSR/MSE=0.011197/0.000649=17.25 Since the calculated F statistic exceeds the critical F value, reject the null hypothesis. 4. The coefficient of determination of 68.3% indicates that variation in the independent variable (debt-to-income ratio) explains 68.3% of the variation in the dependent variable (loan rate). The F test confirms that the slope coefficient is statistically different from zero at the 5% significance level. [LOS 7.e]
Anh Liu is an analyst researching whether a company's debt burden affects investors' decision to short the company's stock. She calculates the short interest ratio for 50 companies and compares this ratio with the companies' debt ratio. Liu estimates a simple regression to investigate the effect of the debt ratio on a company's short interest ratio. The results of this simple regression, including the analysis of variance (ANOVA), are shown below: Regression Statistics R2 0.0933 SE of estimate 2.7905 Observations 50 Coefficients SE t-Statistic Intercept 5.4975 0.8416 6.5322 Debt ratio(%) -4.1589 1.8718 -2.2219 Based on the tables above, the correlation between the debt ratio and the short interest ratio is closest to: A. -0.3054. B. 0.0933. C. 0.3054.
A. In simple regression, the R2 is the square of the pairwise correlation. Because the slope coefficient is negative, the correlation is the negative of the square root of 0.0933, or -0.3054. [LOS 7.e]
Larmer performs the requested regression based on data obtained by a team of engineers who tested 25 vehicles. He defines the dependent variable as Costs, or maintenance costs (in thousands of EUR) for the last calendar year for a given car, and the independent variable as Odometer, or the distance driven (in thousands of kilometers) at the end of the last completed calendar year.: Y = -1.5243 + 0.0423X n an appendix to his report, Larmer notes that the final car reported by the engineers had an odometer reading of 85,650 kilometers as of the end of the last calendar year. According to Larmer's regression results shown in Exhibit 1, the expected maintenance costs over the prior calendar year of the final car reported by the engineers is closest to: A. EUR 2,099. B. EUR 3,623. C. EUR 5,147.
A. To calculate the expected maintenance costs, substitute the odometer reading of the final car reported into the regression equation: Costsi=b0+b1(Odometeri) The expected maintenance costs (in thousands of EUR) for the final vehicle that was tested is: −1.5243+0.0423(85.65)=2.0987
Larmer performs the requested regression based on data obtained by a team of engineers who tested 25 vehicles. He defines the dependent variable as Costs, or maintenance costs (in thousands of EUR) for the last calendar year for a given car, and the independent variable as Odometer, or the distance driven (in thousands of kilometers) at the end of the last completed calendar year. The following or regression results: coefficients t-stat p-value Intercept -1.5243 -3.012 0.0006 Odometer 0.0423 3.208 0.004 Which of the following is the most accurate statistical inference based on the results shown in Exhibit 1? A. The slope coefficient is statistically different from 0 at the 1% significance level B. A 1 unit increase in the odometer reading causes an expected 0.0423 unit increase in annual maintenance costs C. The intercept coefficient is statistically different from 0 at the 5% significance level, but not at the 1% significance level
A. To interpret the result of a significance test on a coefficient, check whether the p-value is above or below the significance level of the test. At the 1% significance level, the coefficient of the slope is significant because the p-value is reported as 0.004, which is less than 0.01. Note that the purpose of regression models is to plot a linear relationship that minimizes the sum of squared error terms. The regression in this example shows that a 1 unit increase in the odometer reading is associated with an expected 0.0423 unit increase in annual maintenance costs. However, the model's outputs should not be used as the basis for conclusions about what, if any, causal relationship exists between the variables.
An analyst is interested in predicting annual sales for XYZ Company, a maker of paper products. The correlation between company and industry sales is 0.9757. The regression was based on five observations. Which of the following is closest to the value and reports the most likely interpretation of the R2 for this regression? A. The R2 is 0.048, indicating that the variability of industry sales explains about 4.8% of the variability of company sales. B. The R2 is 0.952, indicating that the variability of industry sales explains about 95.2% of the variability of company sales. C. The R2 is 0.952, indicating that the variability of company sales explains about 95.2% of the variability of industry sales.
B. The R2 is computed as the correlation squared: (0.9757)^2 = 0.952. The interpretation of this R2 is that 95.2% of the variation in Company XYZ's sales is explained by the variation in industry sales. The independent variable (industry sales) explains the variation in the dependent variable (company sales). This interpretation is based on the economic reasoning used in constructing the regression model. (LOS 7.d)
Jane Sinclair is an equity analyst of Bentall Securities. Sinclair decides to perform a simple linear regression analysis with a country's observed GDP growth for 20X6 as the independent variable (X) and the return on the country's benchmark stock index for the same period as the dependent variable (Y). She gathers the following data: Exhibit 1: Data from selected countries for 2016 Country GDP growth (X) Equity return (Y) Switzerland 0.99% -4.00% Australia 2.87% 11.70% New Zealand 2.77% 19.30% United States 1.58% 9.50% Germany 1.75% 3.50% France 1.33% 6.00% Japan 0.51% 2.70% Average 1.69% 6.96% Variance 0.0000764 0.0055393 After running the regression, Sinclair obtains values of -4.61% for the intercept and 6.863 for the slope. The regression residual for France is closest to: A. -0.96%. B. 1.48%. C. 2.44%.
B. The regression residual for a single data point is the difference between the actual value of the dependent variable (Y) and the value predicted by the regression model. In this example, the actual return on French equities is 6.00%. The predicted value is: −4.61%+(6.863×1.33%)=4.52% Therefore, the regression residual is: 6.00%−4.52%=1.48%
An analyst is examining the annual growth of the money supply for a country over the past 30 years. This country experienced a central bank policy shift 15 years ago, which altered the approach to the management of the money supply. The analyst estimated a model using the annual growth rate in the money supply regressed on the variable (SHIFT) that takes on a value of 0 before the policy shift and 1 after. She estimated the following: Coefficients SE t-Stat. Intercept 5.767264 0.445229 12.95348 SHIFT -5.13912 0.629649 -8.16188 The interpretation of the intercept is most likely the mean of the annual growth rate of the money supply: A. after the shift in policy. B. before the shift in policy. C. over the enter entire period.
B. In a simple regression with a single indicator variable, the intercept is the mean of the dependent variable when the indicator variable takes on a value of zero, which is before the shift in policy in this case.
Anh Liu is an analyst researching whether a company's debt burden affects investors' decision to short the company's stock. She calculates the short interest ratio for 50 companies and compares this ratio with the companies' debt ratio. Liu provides a number of statistics below: Debt Ratio, Xi Short Interest Ratio, Yi Sum 19.8550 192.3000 ∑ni=1(Xi−X¯)^2=2.2225 ∑ni=1(Yi−Y¯)^2=412.2042 ∑ni=1(Xi−X¯)(Yi−Y¯)=−9.2430 Based on the table above, the sample covariance is closest to: A. -9.2430. B. -0.1886. C. 8.4123.
B. The sample covariance is calculated as ∑ni=1(Xi−X¯)(Yi−Y¯)/(n−1)=−9.2430/49=−0.1886
Before running his regression model, Hodgson lists two of the underlying assumptions for linear regression analysis: Assumption 1: The variance of the error term is equal to zero Assumption 2: The independent variable is not random What is the most accurate assessment of Hodgson's description of the assumptions that must be made when performing linear regression analysis? A. Assumption 1 and Assumption 2 are both correct B. Assumption 1 is correct and Assumption 2 is incorrect C. Assumption 1 is incorrect and Assumption 2 is correct
C. Assumption 1 is incorrect. It is assumed that the error term is uncorrelated across all observations. This is known as the homoskedasticity assumption and it implies that the variance of the error term is the same for all observations. It does not imply that this measure has an expected value of zero. Assumption 2 is correct. The assumption of linearity states that the dependent and independent variables are linearly related. This implies that the independent variable is not random. If it were, there would be no linear relationship with the dependent variable. In practice, it may be unrealistic to assume that a chosen independent variable is truly not random. However, it is still possible to rely on the regression results in such circumstances as long as the error term of the regression is uncorrelated with the independent variable. [LOS 7.c]
Kyle Hodgson is an equity analyst covering the automotive parts sector for Frontier Capital. For his latest project, Hodgson uses a simple linear regression model to determine the sensitivity of Cranston Automotive (CRA) stock returns to the overall equity market, which is the beta measure. Hodgson gathers 48 months of return data for CRA, the dependent variable, and a benchmark equity index, the independent variable. Given his chosen model, Hodgson is least likely to observe the: A. fitted parameter for alpha. B. estimated parameter for beta. C. population parameter for beta.
C. Linear regression analysis computes a line that provides the best fit for the observed values of the independent and dependent variables. However, the regression coefficients (intercept and slope) are only estimates of the population parameters, which cannot be observed. The coefficients generated by a linear regression analysis are referred to as estimated parameters or fitted parameters.
Jane Sinclair is an equity analyst of Bentall Securities. Sinclair decides to perform a simple linear regression analysis with a country's observed GDP growth for 20X6 as the independent variable (X) and the return on the country's benchmark stock index for the same period as the dependent variable (Y). The sum of squared residuals calculated is 0.0011631. The standard error of the estimate for the linear regression model is closest to: A. 0.01163. B. 0.04403. C. 0.04823.
C. SEE=√(0.011632/(7−2))=0.048232 Note that 0.04403 is the resulting SEE if the degrees of freedom is n−1 (which is 6) rather than n−2 (which is 5). [LOS 7.e]
An analyst runs a regression by collecting monthly data on the independent variable and the dependent variable. Selected regression output is presented below. A 1% level of significance is used in the tests. Coefficient Standard Error Intercept 0.0095 0.0078 Independent 0.2354 0.0760 Critical t-values for a 1% level of significance: One-sided, left side: -2.441 One-sided, right side: +2.441 Two-sided: ±2.728 Based on the regression output, the analyst should most likely reject the null hypothesis that: A. the slope is less than or equal to 0.15. B. the intercept is less than or equal to zero. C. the independent variable does not explain the dependent variable.
C. The independent variable explains the dependent variable if the slope coefficient is statistically different from zero. The slope coefficient is 0.2354, and the calculated t-statistic is t=(0.2354−0)/0760=3.0974 This statistic falls outside the bounds of the critical values of ±2.728. Therefore, the analyst should reject the null hypothesis that the independent variable does not explain the dependent variable, because the slope coefficient is statistically different from zero. A is incorrect because the calculated t -statistic for testing the slope against 0.15 is: t=(0.2354−0.15)/0.0760=1.1237 This is less than the critical value of +2.441. B is incorrect because the calculated t-statistic is: t=(0.0095−0)/0.0078=1.2179 This is less than the critical value of +2.441. [LOS 7.f]
Which of the following is least likely a necessary assumption of simple linear regression analysis? A. The residuals are normally distributed. B. There is a constant variance of the error term. C. The dependent variable is uncorrelated with the residuals.
C. The model does not assume that the dependent variable is uncorrelated with the residuals. It does assume that the independent variable is uncorrelated with the residuals. [LOS 7.c]
Gabrielle Fox, a hedge fund manager, is preparing an annual performance report for her fund's limited partners. Her analysis is based on the fund's monthly returns over the previous 16 months. Although her compensation is based on an absolute benchmark, Fox has included a section in her report comparing the fund's performance relative to a popular broad market index. As part of her analysis, Fox has performed a linear regression using the monthly return on the index as the independent variable and the fund's return as the dependent variable. ox calculates a t-statistic of -2.394 for the correlation between the returns for her fund and the index the concludes that, using a 5 percent level of significance, this relationship is not statistically significant. However, she acknowledges that she would be more confident in this conclusion if it was based on a larger number of observations. Fox commits to using at least 24 months of returns for any analysis that appears in future reports. Based on her stated criteria, Fox's conclusion regarding her fund's performance relative to the index is most likely: A. a correct decision. B. an example of a Type I error. C. an example of a Type II error.
C. With 16 observations, there are 16 - 2 = 14 degrees of freedom. Since the magnitude of Fox's t-statistic (-2.394) for this relationship is greater than the appropriate test statistic (2.145), Fox should reject the null hypothesis that the correlation is zero and conclude that the relationship is statistically significant. By failing to reject a false null hypothesis, she has committed a Type II error.
Jill Batten is preparing to write a research report on Stellar Energy Corp. common stock. She wants to study the relationships between Stellar monthly common stock returns and the previous month's percentage change in the US Consumer Price Index for Energy (CPIENG). To do this, Batten runs a regression analysis using Stellar monthly returns as the dependent variable and the monthly change in CPIENG as the independent variable. The following tables display the results of this regression model. Coefficients SE t-Statistic Intercept 0.0138 0.0046 3.0275 CPIENG (%) -0.6486 0.2818 -2.3014 Based on the regression, if the CPIENG decreases by 1.0%, the expected return on Stellar common stock during the next period is closest to: A. 0.0073. B. 0.0138. C. 0.0203.
C. From the regression equation, Expected return=0.0138+(−0.6486×−0.01) =0.0138+0.006486=0.0203
Note the following assumptions of regression analysis: Assumption 1 - The error term is uncorrelated across observations. Assumption 2 - The variance of the error term is the same for all observations. Assumption 3 - The dependent variable is normally distributed. Which of the assumptions above is most likely incorrect? A. Assumption 1 B. Assumption 2 C. Assumption 3
C. The assumptions of the linear regression model are that (1) the relationship between the dependent variable and the independent variable is linear in the parameters b0 and b1, (2) the residuals are independent of one another, (3) the variance of the error term is the same for all observations, and (4) the error term is normally distributed. Assumption 3 is incorrect because the dependent variable need not be normally distributed.
Espey Jones is examining the relation between the net profit margin (NPM) of companies, in percent, and their fixed asset turnover (FATO). He collected a sample of 35 companies for the most recent fiscal year and fit several different functional forms, settling on the following model: ln(NPMi)=b0+b1FATOi Regression results : Coefficients SE t-Statistic p-Value Intercept 0.5987 0.0561 10.6749 0.0000 FATO 0.2951 0.0077 38.5579 0.0000 he predicted net profit margin for a company with a fixed asset turnover of 2 times is closest to: A. 1.1889%. B. 1.8043%. C. 3.2835%
C. The predicted natural log of the net profit margin is: ln(y^)=0.5987+(2×0.2951)=1.1889 The predicted net profit margin is: y^=e(1.1889)=3.2835 [LOS 7.h]
A violation of the homoskedasticity assumption will lead to...
Indication that the data series may come from two different regimes. [LOS 7.c]
A violation of the normality assumption will lead to...
Invalid test statistics of the regression coefficients (given a small sample size). [LOS 7.c]
SST (Sum of squared total)
SST = SSE + SSR The total variation in Y is the sum of the unexplained variation in Y and the explained variation in Y [LOS 7.d]
A study was conducted by the British Department of Transportation to estimate urban travel time between locations in London, England. Data was collected for motorcycles and passenger cars. Simple linear regression was conducted using data sets for both types of vehicles, where Y = urban travel time in minutes and X = distance between locations in kilometers. The following results were obtained: Passenger cars: Y=1.85+3.86X, R2 = 0.758 Motorcycles: Y=2.5+1.93X, R2=0.676 The estimated increase in travel time for a motorcycle commuter planning to move 8 km farther from his workplace in London is closest to: A. 31 minutes. B. 15 minutes. C. 0.154 hours.
The slope coefficient is 1.93, indicating that each additional kilometer increases travel time by 1.93 minutes: 1.93 × 8 = 15.44 (LOS 7.b)
An analyst is interested in predicting annual sales for XYZ Company, a maker of paper products. The correlation between company and industry sales is 0.9757. The regression was based on five observations. The calculated intercept is -94.88 and the slope coefficient is 0.2796. Based on the regression results, XYZ Company's market share of any increase in industry sales is expected to be closest to: A. 4%. B. 28%. C. 45%.
The slope coefficient of 0.2796 indicates that a $1 million increase in industry sales will result in an increase in firm sales of approximately 28% of that amount ($279,600). (LOS 7.b)
In the ABC regression example, the estimated slope coefficient was 0.64 and the estimated intercept term was -2.3%. Interpret each coefficient estimate.
The slope coefficient of 0.64 can be interpreted to mean that when excess S&P 500 returns increase (decrease) by 1%, ABC excess return is expected to increase (decrease) by 0.64%. The intercept term of -2.3% can be interpreted to mean that when the excess return on the S&P 500 is zero, the expected return on ABC stock is -2.3%.
Standard error of the slope coefficient
The standard error of the slope coefficient is the ratio of the model's standard error of the estimate to the square root of the variation of the independent variable: [LOS 7.f]
Mean Squared Error (MSE)
The sum of squares error divided by the error degrees of freedom, which are n − k − 1. In simple linear regression, n − k − 1 becomes n − 2. [LOS 7.e]
