Week 4- Logistic Regression
Two different ways to answer the question whether your model fits the data...
1. One is to get a statistic that measures how well you can predict the dependent variable based on the independent variables (measure of predictive power). This are mainly R2 and pseudo-R2 indices (Cox-Snell R2 and Nagelkerke R2). 2. The other approach to evaluating model fit is to compute a goodness-of-fit statistic (Deviance, χ2 and Hosmer-Lemeshow).
Maximum Likelihood estimator
For logistic regression, we use another estimator which is widely used for more complex analysis: maximum likelihood. =>This estimator maximizes the likelihood function.
Linearity in Logistic Regression
If the interaction between predictor and the natural log of the predictor is NOT significant, we DO NOT have a problem with linearity. If it IS significant, we DO have a problem with linearity.
Predicting the probability of y using a s-shaped function (logit transformation)
In more details, we cannot display the actual value of the dependent variable anymore, as the actual values are only 0 or 1. However, we can compute/plot the probabilities for any x value to be 1. You can think that these odds are the change betting on a horse in a horse race. Bet 3:1 = 3/4 ÷ 1/4 = 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 h𝑜𝑟𝑠𝑒 𝑤𝑖𝑛𝑖𝑛𝑔 ÷ 𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 h𝑜𝑟𝑠𝑒 𝑙𝑜𝑜𝑠𝑖𝑛𝑔
SPSS Output- A) Pseudo R²
Predictive power of your model can be assessed with pseudo R2. Comminly, we are using Cox & Snell and also Nagelkerke in SPSS. Higher values are better, but there is no cut of what a high enough value is.
Definition Summaries
Probability = the number of times the event occurs divided by the number of times the event could occur (possible values range from 0 to 1) Odds = the probability that an event will occur divided by the probability that the event will not occur: probability(success) / probability(failure) Odds ratio = the ratio of the odds of success for one group divided by the odds of success for the other group: (probability(success)A / probability(failure)A ) / ( probability(success)B / probability(failure)B ) Log odds = the natural log of the odds
Probability- Definition
Probability is defined as the quantitative expression of the chance that an event will occur. More formally, it is the number of times the event "occurs" divided by the number of times the event "could occur". For a simple example, let's consider tossing a coin. On average, you get heads once out of every two tosses. Hence, the probability of getting heads is 1/2 or .5.
What is the difference between logistic and linear regression?
The first difference to linear regression is that the scale type of the predictor is not interval scaled but dichotomous (i.e., 0 1; male female). (There are other kinds of logistic regressions (e.g., poisson-regression) nut there are beyond of the scope if this lecture.
Odds- Definition
The odds of an event happening is defined as the probability that the event occurs divided by the probability that the event does not occur.
Odds- Definition (EXAMPLE)
The odds of an event happening is defined as the probability that the event occurs divided by the probability that the event does not occur. For example, in a coin-tossing example, the probability of getting heads is .5 and the probability of not getting heads (i.e., getting tails) is also .5. Hence, the odds are .5/.5 = 1. Note that the probability of an event happening and its compliment, the probability of the event not happening, must sum to 1. Now let's pretend that we alter the coin so that the probability of getting heads is .6. The probability of not getting heads is then .4. The odds of getting heads is .6/.4 = 1.5. If we had altered the coin so that the probability of getting heads was .8, then the odds of getting heads would have been .8/.2 = 4. As you can see, when the odds equal one, the probability of the event happening is equal to the probability of the event not happening. When the odds are greater than one, the probability of the event happening is higher than the probability of the event not happening, and when the odds are less than one, the probability of the event happening is less than the probability of the event not happening. Also note that odds can be converted back into a probability: probability = odds / (1+odds).
SPSS Output- B) Goodness of Fit
Χ² Hosmer-Lemeshow
How To Compute A Logistic Regression- B) Compute Regression
=>Log-likelihood: Log-likelihoods tell us how much unexplained variance there is in our model (large values are bad). =>R-statistic: As Wald stats is not reliable, we focus on Nagelkerke (there are others such as Cox & Snell, Or Nagelkerke =>Chi square values =>Deviance test: -2LL =>Main findings: Regression coefficients
Summary of steps of how to Compute A Logistic Regression
=>Screening =>Analysis and model testing =>Check Assumptions =>Interpretation and write up
SPSS Output- Block 0
Block 0: This is a model without including any predictor. Here, we assume simply that all participants are happy (as they are in the majority). Thus we can see that our overall prediction is 54.1%. This is the value we want to beat if we include predictors!
SPSS Output- Check 3: Outliers
Include Cook's distance
Odds ratio- Definition
It is the ratio of two odds. =>Let's say we have males and females who want to join a team. Let's say that 75% of the women and 60% of men make the team. So the odds for women are .75/.25 = 3, and for men the odds are .6/.4 = 1.5. The odds ratio would be 3/1.5 = 2, meaning that the odds are 2 to 1 that a woman will make the team compared to men.
Main assumption of linear regression vs logistic regression
Linear Regression That the association between predictor and outcome is linear, i.e., higher values in x are related to higher (or lower) values in y. Logistic Regression The assumption of linearity is clearly violated in a logistic regression because the value of the outcome is dichotomous (i.e., can only be either 0 or 1). As you can see, there is not data between the values 0 and 1 (outcome variable) and so there is no linear relationships between predictor and outcome. However, we want to keep it simple and use the same ideas as we used in linear regression and use 'a trick' to do so. Instead of predicting the value of y, we predict the probability of y and use a s-shaped function (logit transformation) for that.
Log odds- Definition
Log odds: Another term that needs some explaining is log odds, also known as logit. Log odds are the natural logarithm of the odds. The coefficients in the output of the logistic regression are given in units of log odds. Therefore, the coefficients indicate the amount of change expected in the log odds when there is a one unit change in the predictor variable with all of the other variables in the model held constant.
What can you use logistic regressions for?
Logistic regressions are very useful in general. We can use logistic regressions to predict, for example, the risk of developing a disease (versus not developing it). Or we could use variety of predictors (e.g., race, education, gender and education) to predict which voters vote for republicans or democrats. Last but not least, we will see later that logistic regression are the building block for artificial intelligence trying to mimic a neural network. Thus, logistic regressions are affecting our lives right now...
SPSS Output- Check 2: Independence of Errors
Overdispersion Goodness of fit (chi square) / df Should be ≤ 2
Equation for logistic regression
The way to go around this problem is to use an s-shaped function (i.e., response function) so that we can assess the non-linear relationship in a linear way. Using the logarithmic transformation enables us to predict the dichotomous outcome using our standard linear model which we have discussed before. However, instead of predicting the value of the dependent variable, we predict the probability of y occurring [thus, P(Y) instead of ŷi] using the following equation for logistic regression: As you see, not that much has changed - except that we have now e raised to the minus power of our linear equation. You also can see that the results can only be between 0 and 1. If e raised to the minus power of our linear equation is 0, then we have 1/1, which is 1; whereas if the e raised to the minus power of our linear equation is huge (let's say 1000), the results is approaching 0 (1:1000=0.001).
Z-Statistics (Wald Statistics)
Wald Statistics is similar to the t-statistics for linear regression. But Wald is not very reliable, especially if B is large.
