Research Methods - Quiz 3
Question E (What do the results mean?) pertains to what elements of the article?
"Evaluate and interpret" results and implications See page 285-286 for Meline's checklist.
Question D (What was found?) pertains to what elements of the article?
"Summary of data collected and the statistical or data analytic treatment used." Do the results address *practical significance* and clinical importance? --> meaning does it have a practical application?
The QUALITY of a research design is based on what two components?
(a) *Ability to answer a research question* ...said another way, having a well-constructed hypothesis; (b) *Good Internal Validity*...said another way, having control of extraneous variables
What are three types of peakedness or kurtosis?
(a) *Mesokurtic* = The peakedness of the normal curve. (meso = middle) (b) *Leptokurtic* = sharper shape, meaning more scores at the center (less toward the tails); (think lept is past tense of leap) (c) *Platykurtic* = flatter slope, meaning less scores at center but more scores toward the tails. (think plat rhymes with flat)
What are the four COMPONENTS of a RESEARCH PLAN?
(a) Identify the population of interest; (b) Develop a sampling protocol; (c) Select a design that answers the research question; (with best control of extraneous variables) (d) Choose appropriate statistical tests.
What are two broad sources of information for evidence-based practice?
(a) raw evidence-information that has not been subjected to expert review; (b) pre-filtered evidence-information that has been reviewed by experts, as is the case for articles in peer-reviewed journals.
What are two ways we find information?
*Old-school*: "hand-search" Problems: slow and error-prone. *Modern*: computerized search engines search databases using keywords (index terms or descriptors)
What is *power* and what is another term for power?
*Power* is the ability of a research study to detect significant treatment effects when they are present (also known as: *design sensitivity*). In other words, sensitivity is how good we are at detecting whether an affect is present We also want to consider how good we are at finding whether an affect is not present (not clinically significant) - so we want a Goldilocks amount of sensitivity.
What is *sample size*?
*Sample size* is the total number of participants in a study.
What are some possible SHAPES for frequency distributions?
-- Bell curve vs. skewed curves -- Kurtosis (or the sharpness of the peak or "peakedness")
Describe information literacy tenant #5: Ethically and legally access and use information.
--> Avoid plagiarism by appropriately summarizing, paraphrasing, quoting and acknowledging sources. --> Legally obtain, store, and use text and data, including sound and images. --> Select and consistently use a citation style appropriate to the discipline. --> Cite correctly printed, multimedia, and online sources. --> Distinguish between free and fee-based access to information.
Question B (How does the study fit into what is already known?) pertains to what elements of the article?
--> Beginning of actual article --> Introduction of the problem --> Presentation of background information ("literature review") --> Purpose and rationale In evaluating this section of an article, need to ask: Does this study provide new knowledge? Does it contribute to what we know and don't know? etc.
Describe information literacy tenant #2: Access information effectively and efficiently.
--> Choose key concepts or terms appropriate to the retrieval system selected. --> Recognize that the organization of literature differs by discipline. --> Search the library OPAC, article databases, and web sites fluently, navigating between print and online sources as necessary. --> Follow a citation and connect citation components with searches. --> Locate information remotely and physically by utilizing URLs, call numbers, linking software, and interlibrary loan. --> Modify the search strategy as necessary
Describe information literacy tenant #1: Define the research topic and need for information.
--> Demonstrate a clear understanding of the assignment or information need and its requirements. --> Develop a purpose statement and a timeline for completion of the project. --> Clearly articulate a focused research question or problem. --> Identify types of materials (journals, government publications, books, web presentations) that may be used to complete the research. --> Recognize gaps in information or that information may be limited on the topic.
Describe information literacy tenant #3: Evaluate information critically.
--> Determine if the information discovered is relevant for the needs of the assignment. --> Distinguish between scholarly and popular sources, primary and secondary sources, and mainstream and alternative sources of information. --> Examine and compare information found in books, articles, and web sites, and evaluate for reliability, validity, accuracy, authority, scope, and timeliness. --> Identify prejudice, bias, deception, or manipulation.
Describe information literacy tenant #4: Organize, synthesize, and communicate information for a specific purpose.
--> Manage and store search strategies and search results from a variety of resources using various technological tools. --> Integrate new information with previous information to create knowledge appropriate to answering the research question. --> Present information in a manner that supports the assignment or information need.
Question A (What is the report about?) pertains to what elements of the article?
--> See article's title, keywords, abstract, and statement of purpose. --> These items are used in databases. --> Title should be able to stand on it own --> Abstract is a brief but comprehensive summary of the article.
Why is RANDOMIZATION important?
1. Adds validity to statistical tests 2. Minimizes confounding 3. Reduces biases 4. Supports ethical aspects of research
What are three applications for information literacy?
1. COMD 4800 and many, if not all, COMD classes 2. Reading and evaluating research for academic outcomes; 3. Reading and evaluating research for clinical outcomes, or what has become known as "evidence-based practice"
What are Law's eight steps for solving clinical problems using evidence-based practice?
1. Clearly identify the clinical problem. 2. Gather information from research studies about this problem. 3. Ensure that you have adequate knowledge to read and critically analyze research studies. 4. Decide if a research article or review is relevant to your clinical problem. 5. Summarize the information so that it can be easily used in your practice. 6. Define the expected outcomes for the children [or adults] and their families. 7. Provide education and training to implement the suggested change in practice. 8. Evaluate the practice change and modify (if necessary).
What are the five tenants of information literacy from CSULA with attribution to ACRL?
1. Define the research topic and the need for information ("nature and extent...needed") 2. Access information effectively and efficiently 3. Evaluate information critically 4. Organize, synthesize, and communicate information for a specific purpose 5. Ethically and legally access and use information
What are Greenhalgh and Taylor's eight questions for evaluating *qualitative* research studies?
1. Did the paper describe an important problem addressed via clearly formulated questions? 2. Was a qualitative approach appropriate? 3. How were the setting and the subjects selected? 4. What was the researchers' perspective, and has this been taken into account? 5. What methods did the researcher use for collecting data - and are these described in enough detail?
What are THREE SAMPLING METHODS?
1. Simple random sampling 2. Convenience/accidental sampling 3. Stratified sampling
What are the five characteristics of *normal distribution*?
1. Unimodal (one mode [peak] in the center) 2. Symmetrical: 50% of scores above the center; 50% of scores are below the center 3. Scores run continuously from center to each tail 4. The curve is *asymptotic*: it gets progressively closer to the horizontal axis but never reaches it as moves from center 5. Implied from above: most of the scores are in the center of the curve with progressively lesser at the edges (tails) of the curve
Greenhalgh and Taylor's eight questions cont.
6. What methods did the researcher use to analyze the data - and what quality control measures were implemented? 7. Are the results credible, and if so, are they clinically important? 8. What conclusions were drawn, and are they justified by the results?
What is a *representative sample*?
A REPRESENTATIVE SAMPLE "...includes individuals from each constituency in the target population, including women and ethnic minorities"
What is a distribution?
A distribution is a pattern of scores.
What are the five components of research design?
A plan that includes protocols for: (a) Selecting participants; (b) Controlling extraneous variables; (c) Implementing treatments; (d) Observing variables; and (e) Ensuring ethical procedures.
Summarize the impact of sample size on alpha and beta errors.
A sample size that is TOO LARGE will lead to more chance of an ALPHA error - if it is TOO SMALL, it will lead to a greater chance of a BETA error. Too large = more chance for false positive (you detect relationship that isn't there) Too small = more chance for a false negative (you miss relationship that is there)
A good research design should include a hypothesis. What is a hypothesis?
A tentative explanation for an observation, phenomenon, or clinical problem - under investigation.
What is a *z-score*?
A z-score is a raw score converted to "standard deviation" units. Provides indication of individual's location relative to the mean for the group. z = x - mean ÷ standard deviation
How do you know whether to accept or reject the null given the critical value and test statistic?
Accept the null hypothesis if the test statistic falls within the central (larger) region of the standard curve. Reject the null hypothesis if the test statistic falls within the edge (smaller) region(s) of the standard curve. --> This occurrence means that it is unlikely that your sample would have the test statistic it does if the null hypothesis were true.
Describe the hypothesis testing process STEP TWO: Set an acceptable level of risk.
After setting the hypothesis in step 1, step 2 is to set an acceptable level of risk. There is always an inherent risk, during hypothesis testing, of making a type 1 or type 2 error, and either accepting or rejecting the null hypothesis when it is not accurate to do so. Risk is usually set at 0.05, so 5 out of 100 sample groups tested will possess a sample mean or test statistic that appears statistically significant when there is no statistical significance.
What is kurtosis?
Another variant to normal curve shape is based on the degree of *sharpness of the peak* ("peakedness") or in scientific terms, its *kurtosis.*
What is the ACRL?
Association of College and Research Libraries
Of the two error types, which one can be specified PRIOR to the study?
By convention, amount of Type I error or *α error (alpha) is specified prior to study*. Type II error or β error (beta) can't be specified prior to the study but *it will be inverse to Type I error.*
What is the NULL HYPOTHESIS?
By very formal convention, the hypothesis stating that the research question is false is called the *null hypothesis* (usually shown as H0); in most studies, this translates into: *there is no difference* between conditions; or, there is no significant relationship between the two variables.
How can discipline types affect research variables?
Classically, some disciplines have "cleaner" variables for IVs and DVs and therefore, have true experiments --> (think of most of "hard" sciences; agriculture, etc). Social Sciences, Education, and COMD don't classically have capacities for true experimentation (because our variables are harder to isolate) but we can complete *"controlled inquiry."*
When would you use a one-tailed vs. two-tailed test?
Depending on the experimental question, the critical value may or may not have a direction of outcome, so statistical analysis has to accommodate for this...using one-tailed (more powerful) or two-tailed tests. One-tailed - testing > < relationship Two-tailed - testing EQUAL or NOT EQUAL relationship
How does directionality of outcome impact the critical value?
Depending on the experimental question, the critical value may or may not have a direction of outcome. Example: if the experimental question was: Do children who are dysfluent have normal intelligence (IQ)? The direction of the outcome (yes, they do; no, they don't) is not known, so a two-tailed analysis is appropriate. H0 would be: The IQ of dysfluent children = the IQ of non dysfluent children HA would be: they're different
Describe the hypothesis testing process STEP FOUR: Determine the critical value.
Determination of the critical value is done with the use of a chart (different for one-tailed and two-tailed tests), the alpha significance level (usually 0.05), and the degrees of freedom (n- 1). The critical value is the cut-off point that separates the regions that determine whether the null hypothesis is accepted or rejected.
Describe some example probabilities.
Example probabilities: p > 0.05 "probability is greater than .05" --> this means that there is a greater than 5% possibility that results are due to chance --> ACCEPT NULL p < 0.05 "probability is less than .05" --> this means that chance alone would be the explanation less than 5% of the time --> REJECT NULL
What are *extraneous variables*?
Extraneous variables = uncontrolled rival hypotheses to explain any change in DV
Why and how might you transform a z-score to a normal score?
For ease of knowledge-sharing on clinical tests, z scores can be transformed into a standard score with a mean of 100 and a SD of 15 by: Transformed score = (15) (z) + 100
From what roots does the word "hypothesis" originate?
From Greek: huputithenai = "to place under"...proposal, supposition "Hupu" or "hypo" originally meant and in some uses still means "under" (anatomy), OR "abnormally low" or "reduced" (physiology).
Question C (How was the study done?) pertains to what elements of the article?
Generally known as "Methods" section, three subsections (typically): A. Description of participants or subjects (how selected; how assigned; control group? Paid? demographic characteristics; attrition; sample size?) B. Description of apparatus and material (especially if nonstandard) C. Description of the procedure
Generally, what leads to *greater* POWER in a study?
Generally, the *larger the sample size*, the greater the power of a study (although this is not the only factor that influences the power/design sensitivity)
Define *hypothesis*.
Hypothesis (singular), Hypotheses (plural) = Statements that describe a proposed relationship between two variables, usually with the outcome that the proposed relationship is *either accepted or rejected* (only these two choices of action: a binary decision choice).
Describe the hypothesis testing process STEP SIX: Make the decision about the hypothesis (ACCEPT or REJECT).
If test statistic value exceeds the critical value, reject H0 (and then accept the alternative hypothesis). If test statistic value does not exceed the critical value, must accept H0 (and then must reject the alternative hypothesis)
Selecting participants is a critical aspect for all experiments. Why is participant selection such an important step?
Important for validity and generalizability. Important to establish and clearly state what participant selection criteria were.
What is the role of the hypothesis in research?
In a scientific experiment (research), a research question is re-stated as a hypothesis; data are prospectively collected that are then used to "test the hypothesis," that is, to *choose to accept or reject it.* This hypothesis test is completed with statistics and stated in probability terms of *how much simple chance could have influenced the outcome*...this is another way of defining error, as the likelihood that the obtained measures are accurate estimates of the true measure of the population
Why do we *estimate* population instead of gathering exact population data?
In inferential statistics, emphasis is on estimation of population parameters because the "population" is too large and too inaccessible...so, the only reasonable procedure is to take a random sample and make an estimate based on that sample. This is the only practical approach.
Why not conduct the study on the ENTIRE target population?
It is not possible to observe all members of the target population, so a SAMPLE (a sub-set of the population) is taken out of the population - this sample is intended to be representative of the entire population. The hope/logic then becomes, the observations on the sample are generalized to population as a whole.
What are *parameters*?
Knowledge and assumptions about the distribution of the populations are called its "parameters."
Who is the mathematician most associated with normal distribution?
Mathematician most associated with normal distribution is De Moivre who looked at the basic probabilities of outcomes.
How would you state the alternative hypothesis?
May be written as: H sub A - it usually states that the mean of the NO TREATMENT GROUP *does NOT equal* the mean of the TREATMENT GROUP. In other words: There IS a difference and that observed difference in the sample is due to a non-random event.
How would you state the null hypothesis?
May be written as: H sub zero - it usually states that the mean of the NO TREATMENT GROUP is *equal to* the mean of the TREATMENT GROUP In other words: There is no real difference, or said another way: any observed differences in the sample are due to a purely chance (random) occurrence.
How does Meline define *distribution*?
Meline defines distribution as a "pattern of scores" or how often a score occurs in a measurement.
How can we use samples to determine population parameters?
Meline points out that all variables have distributions. Statisticians extend this concept to apply it to all statistical tests, such as mean and standard deviation, to get an idea whether, for example, the mean of a sample (drawn from a population) varies significantly from the "true" population mean or whether it varies from the mean calculated from another (different) sample. For example, if we want to find the true population, we would gather multiple samples and find the mean of all those samples.
What five questions, proposed by Locke, Silverman, and Spirduso, does Meline state is applicable to *both qualitative and quantitative* research?
Meline's model has five basic questions: (a)What is the report about? (b)How does the study fit into what is already known? (c)How was the study done? (d)What was found? (e)What do the results mean?
What is the difference between the bell-curve and skewed curve?
Most typical is "bell-shaped" curve (unimodal and symmetrical). Other non-bell-shaped distributions are possible. If scores cluster to one end of the score distribution or to the other end of the score distribution, the result would be asymmetrical to one end or the other. These types of asymmetrical distributions are referred to as: *Skewed* (with more scores toward either tail). --> Positive skew --> Negative skew
Can both the null and alternative hypotheses be true at the same time?
NO. By convention, the null hypothesis (H0) is stated to be the opposite to the experimental question. (mutually exclusive) Therefore, a finding that the null hypothesis (H0) is TRUE means the anticipated outcome of the experimental question was NOT found. However - when the null hypothesis (H0) is FALSE, this means that there has to be some explanation for this outcome - the alternative hypothesis (HA) - which is what the experimenters had proposed in the first place.
Describe a negative skew.
Negative skew: more scores toward right (higher values). Called negative skew because tail point toward the negative.
How do we know our estimates are accurate?
Over the years, statisticians have developed and tested formulas, models, and techniques that enable us to, for example: take a sample, calculate its mean, and estimate with a specified degree of certainty how well this sample mean estimates the "true" mean (the population mean) The logic of this approach in inferential statistics is based on the determination of whether this outcome was due to what would have been the likelihood of this outcome due to chance alone...(chance = probability of a sampling error)
What would characterize poor research designs?
Poor research designs likely characterized by influences of extraneous variables...limiting the causal relationship between the IV and the DV.
When do you use parametric vs. nonparametric statistics?
Populations that have a normal distribution are tested with "parametric" statistics. A population that doesn't have a normal distribution is tested with "nonparametric" statistics. Note: Most studies attempt to use parametric tests.
Describe a positive skew.
Positive skew: more scores toward left (lower values); Called positive skew because tail pointed towards positive
What should be be wary of regarding pre-filtered evidence types?
Regarding pre-filtered evidence, be warned that research published in scientific journals gives the readers some confidence in the scientific credibility of research findings, but scientific credibility does not necessarily mean that the findings represent the truth. Both pre-filtered and raw evidence types require careful evaluation by consumers.
What are the symbols for population vs. sample parameters?
Sample mean = x bar Population mean = μ ("mew") Sample Standard Deviation = S Population Standard Deviation = σ ( sigma) Sample size (number) = n Population size (number) = N
How does sample size affect power?
Sample size (n) also determines (or at least, contributes to): The power of the test - defined here as the probability of rejecting H0 when it is false (one of the correct decisions). The larger the sample size, the more powerful the test of H0 (but recall that sample size is NOT the ONLY thing that determines power) In the context of the above statement, a weak test will more likely accept H0 when it is false - this is Type II error or β error. In other words, the smaller the n, the greater the risk of Type II error or β error.
What does sample size affect or help to determine?
Sample size (n) determines: (a) What probability distribution to use (b) Power
How does sample size affect what probability distribution you use?
Sample size (n) determines: What probability distribution to use: ---> standard normal distribution, or small (in more complex situations, several other distributions are possible --> depending on assumptions that exist or can be made about the population being sampled...high-level statistics)
Describe the hypothesis testing process STEP THREE: Choose the sample size.
Sample size will impact distribution type and power, so it is important to establish the correct sample size to minimize the risk of each error type. We want a sample that is not too large, which could lead to greater chance of alpha error. And we want a sample that is not too small, which could lead to greater chance of beta error.
What is the *standard normal distribution*?
Standard normal distribution is the conversion of scores into standardized (z) scores with a common mean (0) and standard deviation (1)
The hypothesis is a key element for what TYPE of STATISTICS?
Statement of hypothesis is a key element /convention of *inferential statistics*...making conclusions (inferences) about populations from sample data. (Contrast to descriptive statistics: Chapter 4, measures of location and variability).
What are the six steps of the HYPOTHESIS TESTING PROCESS?
Step one: State the hypothesis Step two: Set an acceptable level of risk Step three: Choose the sample size Step four: Determine the critical value Step five: Compute the test statistic Step six: Make a decision about the hypothesis: accept or reject
TRUE OR FALSE: In formal research studies, both the null hypothesis and the alternative or experimental hypothesis are explicitly stated.
TRUE In formal research studies, both the null hypothesis and the alternative or experimental hypothesis are explicitly stated.
What is the TARGET POPULATION?
The *target population* is the population of interest. --> Identifying the population of interest is first component of the research plan. Target population = ALL individuals with a common characteristic (example: ALL adults with aphasia).
How does the capacity to randomize differentiate experiment types?
The capacity to randomize in a study = true experiment. Lack of capacity to randomize = quasi-experiment
What is the *critical value*?
The critical value is a cut-off point: that value of probability (chance occurrence) that divides the decision-making continuum into ACCEPT or REJECT the H0.
What is a good hypothesis?
The foundation of a research design. It must: - Be clearly and concisely stated; - Link two variables thought to be related; - Be testable; - Make sense.
What does it mean when the alpha value is greater?
The greater the value of α, the more likely chance alone could explain the outcome. Therefore, the smaller the value of α, the less likely chance alone could explain the outcome...meaning the findings likely are accurate estimates of "reality."
Describe the hypothesis testing process STEP ONE: State the hypothesis.
The hypothesis is usually stated as an alternative an null hypothesis, where the null is the status quo (nothing happens) and the alternative is what the researcher anticipates happening. The alternative hypothesis, in inferential statistics, should state an anticipated causal relationship between the independent and dependent variable.
Summarize the relationship between sample size and beta error.
The larger the sample size, the more powerful the test of H0 - the greater the possibility of rejecting H0 when it's false (correct). The smaller the n (the weaker the test), the greater the risk of Type II error or β error. So a larger sample size means that you will be less likely to miss an affect (say that there is no effect when there is).
How is the research question stated?
The research question is stated as two mutually exclusive hypotheses: one that is supportive of the research question and another that is NOT supportive of the research question - in fact, states that it is false. i.e. NULL hypothesis = the treatment is NOT effective ALTERNATIVE hypothesis = the treatment IS effective
How does the researcher formulate the research question and hypothesis?
The researcher has the notion that there is a difference between *two conditions* (that an IV has a causal effect regarding the DV). This becomes the basis for the research question (and the research project). In the convention of scientific research, this research question is re-stated as *two mutually exclusive hypotheses:* one that is NOT SUPPORTIVE of the research question - in fact, *states that it is false* (*"there is no difference"*); and another that IS SUPPORTIVE of the research question (*"there is a difference"*).
What is the ALTERNATIVE HYPOTHESIS?
The statement of the expected research outcome is called the *alternative hypothesis* (HA).
How is the test statistic chosen?
The test statistic (t) is chosen based on: a) The hypothesis b) The (assumed) distribution of the target population c) Sample size (n) d) Other characteristics of the sample
Describe the hypothesis testing process STEP FIVE: Compute the test statistic.
The test statistic is calculated based on the hypothesis, assumptions about the population (i.e. normal vs. not normal distribution), sample size, and other characteristics.
What is the definition of *design*?
To conceive or fashion; to formulate a plan; to create or execute in a highly skilled manner.
What are the FOUR POSSIBLE OUTCOMES when testing the null hypothesis?
Two of these play out to be *correct statements*: (a) H0 accepted when H0 is True; (b) H0 rejected when H0 is False. Two of these play out to be *incorrect statements*: (a) H0 rejected when H0 is True... ----> Type I error OR α error (alpha error) ---->...False Positive. (b) H0 accepted when H0 is False... ----> Type II error OR β error (beta error) ---->...False Negative.
Describe a Type 1 vs a Type 2 error.
Type 1: Rejecting the null when the null is true --> AKA alpha error --> False positive --> You are saying there is an effect when there is no effect Type 2: Accepting the null when the null is false --> AKA beta error --> False negative --> You are saying that there is no effect when there is, in fact, an effect
Of the two error types, which does Meline consider to be the more serious?
Type I error "considered the more serious of the two possible errors."
What is the typical Type 1 error level of probably?
Usually, Type 1 error set at 0.05 level of probability...this is conservative and "guards against the likelihood of concluding a treatment works when it may not."
What does it mean when you set alpha error (type 1 error) at 0.001 level of probability?
With Type I (α error) is set to a 0.001 level of probability, this means the researcher risks 1 time out of 1000 that this error in inference could occur simply due to chance (in the measurements).
What does it mean when you set alpha error (type 1 error) at 0.01 level of probability?
With Type I (α error) is set to a 0.01 level of probability, this means the researcher risks 1 time out of 100 that this error in inference could occur simply due to chance (in the measurements).
What does it mean when you set alpha error (type 1 error) at 0.05 level of probability?
With Type I (α error) is set to a 0.05 level of probability, this means the researcher risks 5 times out of 100 that this error in inference could occur simply due to chance (in the measurements)
How do we determine the critical value?
With the selected level of risk in mind (for example: 0.05 or 0.01 or 0.001), we have to take the outcome (a number) from whatever statistical test was applied to the data and go to a table of values...typically, to use the table, we will need to know the sample size (n) of scores to get "degrees of freedom" which will be: df = n-1
After an experiment, what do you do with the null hypothesis?
You can either: --> accept the null hypothesis (H0 is true) --> reject the null hypothesis (H0 is false) You would get to this conclusion by determining whether or not it would be very unlikely to have collected the sample you did if H0 was true.
What is the MAX-MIN-CON principle?
} Maximize the experimental variable (the IV) } Minimize untreated changes in the DV } Control the effect of extraneous variables
