Chap 3: The phoenix of statistics
NHST is a hard habit to break bc it ...
requires many people to make the effort to broaden their statistical horizons
scientists may contribute to publication bias by capitalising on ... to show their results in the most favourable light possible
researcher degrees of freedom
what are Cohen's benchmarks for effect size?
d = .02 (small) d = .05 (medium) d = .08 (large)
what is the formula for Cohen's d?
d = X1 - X2 / s
pearson's correlation coefficient r can vary from ....
- 1 (perfect negative relationship) 0 (no relationship) 1 (perf positive)
pearson's correlation coefficient r is a measure of the strength of a relationship between ...
- 2 continuous variables - one continuous & one categorical variable containing 2 categories
what are the 3 effect sizes that Field discusses?
- Cohen's d - Pearson's correlation coefficient r - the odds ratio
what are other older terms for p-hacking?
- fishing - cherry-picking - data snooping - data mining - data diving - selective reporting - significance chasing
the transparency and openness promotion (TOP) guidelines, is a set of standards for open science principles. They cover guidelines such as ...
- pre-registration of study & analysis protocols - replication transparency with data, analysis scripts, design, analysis plans, research materials - replication
open science initiatives push incentive structures towards ...
- quality not quantity - documenting research (not storytelling) - collaboration (not competition)
there are many ways scientists can influence the p-value, these are known as researcher degrees of freedom & include ...
- selective exclusion of data - fitting different statistical models, but only reporting most favourable results - stopping data collection at a point other than that decided at the study's conception - including only control variables that influence the p-value in a favourable way
what practices represent p-hacking?
- trying multiple analyses & reporting only the one that yields sig results - deciding to stop collecting data at a point other than when predetermined sample size is reached - including (or not) data based on the effect they have on te p-value
the transparency and openness promotion (TOP) guidelines, is a set of standards for open science principles. For each standard, there are levels defined from 0 - 3. What do they represent?
0 - the journal merely encourages data sharing or says nothing 3 - data is posted on a trusted repository & results reproduced independently before publication
what are the 3 common misconceptions about what a statistically significant result allows you to conclude?
1) a significant result means that the effect is important 2) a non-significant result means that the H0 is true 3) a significant result means that the H0 is false
the practice in research articles of presenting a hypothesis that was made after data collection as tho it were made before
HARKing
the ASA released 6 principles for scientists using NHST. The 2nd is that you don't interpret p-values as ...
a measure of the probability that the hypothesis in question is true
what is the 'open science' movement?
a movement to make the process, data and outcomes of research freely available to everyone
one way scientists have looked for evidence of p-hacking is by extracting the reported p-values across a range of studies on a topic, or within a discipline, and plotting their frequency. What does the line show?
a p-curve; the no. of p-values you'd expect to get for each value of p
NHST is the dominant method for testing theories using stats; it is compelling bc it offers ...
a rule-based framework for deciding whether to believe a hypothesis
one common misconception about what a statistically sig result allows you to conclude is that a sig result means that the H0 is false. Why is this not the case?
a sig test statistic is based on probabilistic reasoning, which limits what we can conclude
r quantifies the relationship between 2 variables, thus if one is categorical with 2 groups (coded as 0 and 1), then you get ...
a standardised measure of the difference between 2 means (kinda like cohen's d)
the most important conclusion about p-curves is that it's very difficult to extract enough info to ...
accurately model what p-hacking might look like & thus ascertain whether it happened
NHST is also based on long-run probabilities, i.e. the alpha value (e.g. .05) is a long-run probability & means that ...
across repeated identical experiments the probability of making a type 1 error is .05
other things being equal, effect sizes are not ...., unlike p-values
affected by sample size
why is a 'standardised' measure good? for example with effect sizes
allows you to compare effect sizes across different studies that have measured different variables, or have used different scales of measurement
NHST is also based on long-run probabilities, i.e. the alpha value (e.g. .05) is a long-run probability. That probability does not apply to ...
an individual study where you either have (p = 1) or have not (p = 0) made a type 1 error
the analysis of p-curves looks for p-hacking by examining ...
any p-value reported within and across the articles studied
the ASA published some guidelines about p-values and the ways in which you might address the problems with NHST. The first way to combat the problems is to ..
apply sense when you use it
the p-value is the probability of getting a test statistic ... from an infinite no. of identical replications of the experiment
at least as large as the one observed relative to all possible values of tnull
one common misconception about what a statistically sig result allows you to conclude is that it means that the effect is important. Why is statistical significance not the same thing as importance?
bc the p-value from which we determine significance is affected by sample size e.g. v small effects will be sig in large samples (vice versa)
questionable research practices are not purely the fault of NHST, but NHST nurtures these temptations by fostering ...
black and white thinking, i.e. sig results garner much greater personal rewards than non-sig
altho the sample size doesn't affect the computation of the effect size in the sample, it does affect how ...
closely the sample effect size matches that of the population (the precision)
the peer reviewers openness initiative asks scientists to ...
commit to the principles of open science when they act as expert reviewers for journals
NHST is biased by researchers ...
deviating from their initial sampling frame (e.g. by stopping data collection earlier than planned)
cohen's d is the ...
difference between means divided by the SD
.... add info that you don't get from a p-value
effect sizes
the ASA released 6 principles for scientists using NHST. The 6th id that 'by itself, a p-value doesn't provide a good measure of ...'
evidence regarding a model or hypothesis, i.e. there may be many other hypotheses that are more compatible with the data than the hypotheses you have tested
when using cohen's d, and the groups do not have equal standard deviations, there are 2 common options. The first is to use the SD of the control group or baseline. This option makes sense bc any intervention might be ...
expected to change not just the mean but also the spread of scores - therefore the control group/baseline SD will be a more accurate estimate of the 'natural' SD for the measure you're using
scientists have looked at whether there's general evidence for p-hacking by looking at the distribution of p-values you would expect to get if p-hacking doesn't happen with the distributions you might expect if it does. One example is to ...
extract the reported p-values across a range of studies on a topic, or within a discipline, and plot their frequency
part of the open science movement is about providing ...
free access to scientific journals
researcher degrees of freedom refers to the fact that a scientist ...
has many decisions to make when designing & analysing a study
an empirical probability is the proportion of events that ...
have the outcome in which you're interested in an indefinitely large collective of events
one common misconception about what a statistically sig result allows you to conclude is that a non-significant result means that the H0 is true. Why is this wrong?
if p > .05, then you can decide to reject the H1 - but it doesn't mean the H0 is true. i.e. there may still be an effect, but it just wasn't big enough to be found
the ASA released 6 principles for scientists using NHST. The first is that p-values can indicate how ...
incompatible the data are with a specified statistical model, i.e. we can use the value of p to indicate how incompatible the data is with the H0 (smaller ps = more incompatibility)
another way to see if there's evidence for p-hacking in science is to focus on p-values in ...
individual studies that report multiple experiments
the current incentive structures in science are ....
individualistic rather than collective
what is the crucial point about empirical probabilities?
it applies to the collective (not individual events)
when the group SDs are different, using the pooled estimate can be useful to calculate Cohen's d, however ...
it changes the meaning of d bc we're now comparing the difference between means against all the background noise in the measure
what is perhaps the biggest practical problem created by NHST?
it encourages all-or-nothing thinking, i.e. if p < .05 then an effect is sig, but if p > .05 it is not
effect sizes offer us something potentially less/more misleading than NHST
less
NHST works on the principle that you'll make a type 1 error in 5% of an infinite no. of repeated, identical experiments. The .05 value of alpha is a ...
long-run & empirical probability
when one variables increases and the other decreases
negative relationship
a non-sig result should never be interpreted as ...
no difference between means or no relationship between variables
the ASA released 6 principles for scientists using NHST. The 3rd is that scientific conclusions and policy decisions should ...
not be based only on whether a p-value passes a specific threshold
both p-hacking and HARKing are cheating the system of NHST, bc in both cases it means that you're ...
not controlling the type 1 error rate (bc you're deviating from the process that ensures that it is controlled) & thus you have no idea how many type 1 errors you will make in the long run
pearson's r is ...., so an effect with r = .60, is not twice as big as one with r = .30
not measured on a linear scale
.... help to combat many of the wider problems into which NHST feeds
open science practices
the ASA released 6 principles for scientists using NHST. The 4th is: DO NOT
p-hack
researcher degrees of freedom that lead to the selective reporting of significant p-values
p-hacking
what are researcher degrees of freedom that relate closely to NHST?
p-hacking & HARKing
when using cohen's d, and the groups do not have equal standard deviations, there are 2 common options. The second option is to ..
pool the SDs of the 2 groups (if the groups are independent)
when one variable increases and the other also increases
positive relationship
the ASA released 6 principles for scientists using NHST. The 5th is not to confuse statistical significance with ...
practical importance
HARKing is the practice in research articles of ....
presenting a hypothesis that was made after data collection as tho it were made before
submission of pre-registration of research is reviewed by relevant experts and if the ...., then it's accepted with a guarantee to publish the findings no matter what
protocol is deemed to be rigorous enough and the research question novel enough
significant findings are much more likely (7x) to be published than non-significant ones = ?
publication bias
what are the benchmarks for pearson's r?
r = .10 (small) r = .30 (medium) r = .50 (large)
by guaranteeing publication of the results, registered reports should ...
reduce publication bias and discourage questionable research practices aimed at nudging p-values below the .05 threshold
what are the consequences of the misconceptions of NHST?
scientists overestimate the importance of their effects; e.g. by: - ignore effects that they falsely believe don't exist due to accepting the H0 - pursue effects that they falsely believe exist bc of rejecting the H0
a report by the APA on the limitations of NHST, didn't recommend against NHST, but instead suggested that ...
scientists report things like CIs & effect sizes to help them evaluate findings without reliance on p-values
scientists may contribute to publication bias by ... and excluding non-sig ones
selectively focusing on sig findings
success in science is largely defined by ...
significant results
an effect size is an objective and (usually) ..
standardised measure of the magnitude of observed effect
pre-registration of research can be done in a registered report in an academic journal or more informally. A formal registered report is a ...
submission to an academic journal that outlines an intended research protocol before data collected
Cohen's d is simple enough to compute & understand, but it has 2 small inconveniences. The second is that altho the difference between means gives an indication of the signal, it does not ...
tell us about the 'noise' in the measure
when using cohen's d, and the groups do not have equal standard deviations, there are 2 common options. The first is to use ...
the SD of the control group or baseline
the p-value is the frequency of the observed test statistic relative to all possible values that could be observed in ...
the collective of identical experiments
when computing p-curves (to test p-hacking), whereby the curve is affected by the effect size & the sample size - what happens when the effect size if greater than 1?
the curve has an exponential shape - bc smaller ps (more sig) occur more often than larger ps
when computing p-curves (to test p-hacking), whereby the curve is affected by the effect size & the sample size - what happens when there's no effect (0)?
the curve is flat - all p-values are equally likely
the pearson's correlation coefficient can also be used to quantify ...
the difference in means between 2 groups
scientists have looked at whether there's general evidence for p-hacking by looking at ...
the distribution of p-values you would expect to get if p-hacking doesn't happen with the distributions you might expect if it does
the p-value is affected by ... of the researcher
the intention
pre-registration of research, as part of the open science movement, refers to ...
the practice of making all aspects of your research process publicly available before data collection begins
altho NHST is the result of trying to find a system that can test which of 2 competing hypotheses (H0 or H1) is likely to be correct, it fails bc ...
the sig of the test provides no evidence about either hypothesis
pearson's correlation coefficient r, is a measure of ...
the strength of the relationship between 2 variables
Cohen's d is simple enough to compute & understand, but it has 2 small inconveniences. The first is that the difference in means will be expressed in ...
the units of measurement for the outcome variable (so if it's units of 'positivity' for example, it's less tangible)
incentive structures in science that reward publication of sig results also reward ...
the use of researcher degrees of freedom
what is the problem with using Cohen's benchmarks?
they encourage lazy thinking
the strength of the pearson correlation reflects how ...
tightly packed the observations are around the model (line) that summarises the relationship between variables
one problem with NHST is that significance does not tell us about the importance of an effect. What is a solution to this?
to measure the size of the effect being tested in a standardised way
researchers are engaging in practices ... and that's why these values are over-represented in literature
to nudge their p-values below the threshold of significance
one way to see if there's evidence for p-hacking in science is to focus on p-values in individual studies that report multiple experiments. A low probability means that it's highly unlikely that the researcher would get results which are ...
too good to be true - with the implication that p-hacking has occurred
by having a public record of the planned analysis strategy, deviations from it will be ...
transparent, i.e. p-hacking & HARKing will be discouraged
p-curves can't be completely trusted to pinpoint p-hacking as when you use p-curves that better resemble p-hacking behaviour, the evidence ...
vanishes
Another problem with NHST is that the conclusions from it depend on ...
what the researcher intended to do before collecting data
when is cohen's d favoured over pearson's r?
when group sizes are discrepant, r can be quite biased compared to d
when is pooling the SDs of the 2 groups useful for calculating Cohen's d?
when the group SDs are different
what is an advantage of having your protocol for a study reviewed by experts?
you can get useful feedback before data collection
by using effect sizes we overcome one of the major problems of NHST (bc they're not affected by sample size). But the situation is more complex bc, like any parameter, ...
you will get better estimates of the population value in large samples (vs small)
you should evaluate an effect size within the context of ...
your specific research question