Statistics (EDRE 7200)

• Replicability

- Can someone else do it & reach similar conclusions?

A good Research Question...

- Excites your passion, contributes to the field - Is creative, unique, adaptive, and/or synthetic - Is valuable, manageable, and affordable (sometimes we need to break large questions into smaller ones) - Is doable ethically, logistically, skillfully - Is relevant - has an answer to "so what" question - Does not rely only on methodology we know - Does NOT depend on statistical significance

Validity

- How can we get the most meaningful results? - How do limitations impact results & interpretations?

Skepticism lies at the heart of research

- Ioannidis, J.P.A. (2005). Why most published research findings are false. PLoS Medicine, 2, 696-701. - We need to ask questions about everything that occurs in research, both our own and of others • No matter how trusting you are... don't just believe without questioning and demanding evidence - Don't be afraid of others criticizing or questioning your work • just be ready to defend the decisions you made

An Introduction also usually includes

- Significance of the Study • Importance to theory, practice, policy - Relevant Theory & Empirical Literature • Theory addresses the relationships among variables in an attempt to describe, predict, or explain phenomena - Research Hypotheses • Proposed answer to the research question (based on theory and empirical research literature) - Definitions • Conceptual, theoretical, practical - Delimitations (more related to external validity) • scope of the research, controlled by researcher - Limitations (more related to internal validity) • Elements of study design the researcher cannot control, but can try to manage • Don't want fatal flaws, need limitations that can be managed and will not impact veracity of conclusions

The research question should contain:

- The variables of interest • Independent variables (predictors, potential causes) • Dependent variables (outcomes, effects) • Other variables (e.g., mediating, moderating, control) - The relationship to be studied among the variables • Correlational, predictive, causal - The population of interest - Any relevant, specific context for the study • e.g., laboratory, timing (e.g., season, length), materials

What we'll be talking about: Validity (a.k.a., evils that researchers face) 4 main types, there is a 5th issue of theoretical validity.

-No Theoretical Framework (theoretical validity) • Inappropriate Sampling (external validity) • Poor Instrumentation (construct validity) • Over-reaching conclusions (internal validity) • Analytical concerns (conclusion validity) • Non-Significance (non-important) (return of theoretical validity)

nominal variables

= CATEGORIES Numbers are Synonyms for Names, Like Jersey # Numbers are For Groups, like 1=Goalkeepers And 2=Field Players Statistics have no meaning, just synonyms for the players names When we assign numbers to things: measurement Could also separate by colors, group 1 is goalkeepers and they can use their hands, group 2 cannot. Could have switched the group numbers around and it would not have mattered. The numbers do not have meaning in the numeric sense

seesaw example

A child is sitting on a seesaw. The other end goes up. We need to create balance, so if we add another same size child on the other end we are balanced. The fulcrum is the point of balance. If you add another child on the shoulders of the original child on one side, it is no longer balanced. Is there another way to balance out the second child without adding another child on the shoulders? We would add two more children on the other side and spaced out to create balance. Adding intervals: we have two children on seat 4 on the left and on the right we have a child at 1, 3, and 4. We think of the balance point as 0, anything to the left is negative and the right is positive. Adding the scores: -8 on the left side and then 8 on the right side. These sums are equal and balance out. (positive and negative just mean direction). We can calculate the mean as the balancing point - 0. The distance from the mean are balanced, that is what the mean represents. Mean: sum of scores divided by the number of scores. (0) Median: Balances based on number of cases, not distance. If we needed to calculate from the 5 scores, it would 1. There are two scores above and below. A different way to think about a central tendency Location: Location of the data is based on the center of the data. It can be moved around. It is just based on the center, so if we moved one child, it is the new seat number Mode: not a good one, it is the most common score of the data (-4). Doesn't give a good idea of the center Another child is added and sits on the shoulder of one child on the right side. We needed to add 2 children to the left side so we can create equal weight. Median: Now we have 8 cases, so we can take the two middle points and average them. Distribution: if we have a closely symmetric distribution the mean and median will be close. Mode: 2 modes because there are two children on shoulders. Mean: sum/N. Parametric statistic that we use to talk about central tendency Dispersion/Variation: how spread out are the scores? 5 children on each others shoulders. there is no variation, everyone is at seat 0. We can add deviation scores: how far they are from the mean (deviation - mean) (0 - no variation) we could call this a constant Standard deviation is the square root of variance Now lets say that there are children on shoulders at the ends of the seesaw. Mean and median will be the same because its perfectly symmetric. Deviation is -4 - 0 and 4 - 0 so it is -4 for the left side (it is below the mean) and 4 for the right side (it is above the mean). This is maximum variation, it is not possible for people to be farther from the mean. Everyone must be at the most extreme score above and below the mean. More typical case: mean is still 0, 2 children on seat 1 and -1 and 1 child on seat -2 and 2. Median is 0. Deviation scores sum to 0 because we have equal weight above and below the mean. Now we need to square each deviation scores and then add them up. The sum of squares is 12 (this is a measure of variation). The more cases we add the larger the sum of squares. We then look for the average sum of squares from the mean. 12/5 = on average 2.4 squared deviations from the mean. Variance: sum of squares/N-1. We then take the square root and it becomes the standard deviation. SQROOT of 2.4 = 1.5. It is just descriptive, it says how spread out the scores are. First parameter was the mean and then standard deviation (how spread out from the mean) Range: how far apart are the minimum and maximum scores. 2 - (-2) = 4. Range of these score are 4, it gives us a sense how spread out it is. Outliers will affect this

Outlier

A value much greater or much less than the others in a data set

Basic Designs

Basic designs • Just collect data - Observation (O) - e.g., observational, correlational • Collect data after manipulation or across traits - e.g., non-, quasi-, or true Experimental • Different designs allow us to control threats to internal validity differently - For example, adding random assignment and control groups give us stronger comparisons

Reporting Internal Validity

Clearly indicate in "experimental" designs what interventions, treatments, and/or manipulations were implemented (or what traits compared) • Verify integrity or fidelity of manipulations • Report how potential threats to validity were controlled (and what could not be controlled) • Report how random assignment was used • Describe whether you will use a between subjects or within-subjects design

Need for Control

Decide on your methods of control • "Third variable" problems (covariates) - Sometimes a third variable is responsible for an outcome on the dependent variable (Town in NJ, more churches and more alcoholics- one possible conclusion is that more churches cause more alcoholics, but it was because the population was increasing Why is crime and ice cream connected-they increase in warmer weather. Third variable problem) • Control potentially confounding variables by including them in data collection or design - If left uncontrolled, they are called alternative explanations (extraneous) • Control through design vs. Statistical control

INTERVAL

Equal intervals between numbers but ZERO is arbitrary, or does not mean "ABSENCE OF" We need INTERVAL or RATIO to use parametric statistics meaningfully For example: 0s are not meaningless on the temperature scales, but they do not mean absence of. Every line indicates the same temperature gain everywhere on that scale, important for equal intervals Another example: 0 on emotion scale, does not mean you do not have any emotions at all

RATIO

Equal intervals between numbers but ZERO is meaningful and means "ABSENCE OF Like on rulers 0 height, that is meaningful

Frequency

How many people had those scores = mode.

Variables

How we are getting our measurements for the psychological constructs Sometimes need to infer. Variable - people can differ on constructs

Scewness

Measure of symmetry. If it is 0 that means it is symmetric

• Significance -

NOT statistics... but rather, how important is it to answer your research question? • Not to add more bad research to the literature 1

Sample Size as Element of Design

Pay attention to sample size as a design element • Larger samples can but may not represent the population better (representativeness is key) - The larger the representative sample, the better the results should cross-validate and remain stable - Sample size matters for Statistical Power • Choose n based on larger and correct power analysis - Sample size matters for the accuracy of our estimates (e.g., confidence intervals better with larger samples) - Sample size matters for generalizability and validity

ORDINAL =

RANKING Numbers mean more or less of some trait or variable, with no indication of how much difference between the numbers e.g., Ranking of favorites or items e.g., 1=Small, 2=Medium, 3=Large e.g., 1=Taller, 2=Average, 3=Shorter (could have been ordered in the opposite way, any direction we want. We just need to be aware so we can make note of that. Because there is order, that is why it is ordinal not nominal) Parametric statistics are often not meaningful (e.g., mean) 2018 Tour de France 1 = Winner = Geraint Thomas ( GBR Team Sky ) 2 = Second = Tom Dumoulin ( NED Team Sunweb ) 3 = Third = Chris Froome ( GBR Team Sky ) (we do not know the difference between the times, it could have been really close or really far apart, we just know their ranking) Points = Peter Sagan ( SVK Bora-Hansgrohe ) Mountains = Julian Alaphilippe ( FRA Quick-Step Floors )

conclusion validity

The degree to which conclusions you reach about relationships in your data are reasonable

interquartile range

The difference between the upper and lower quartiles. Subtract 25th from 75th and it will give the middle 50%. It gets rid of extreme values

Lovitts (2005) Component 1: Introduction

The introduction • Includes a problem statement • Makes clear the research question to be addressed • Describes the motivation for the study • Describes the context in which the question arises • Summarizes the dissertation's findings • Discusses the importance of the findings • Provides a roadmap for readers

Lovitts (2005) Component 4: Methods

The research design and methods applied/developed are • Appropriate • Described in detail • In alignment with the question addressed and the theory used In addition, the author demonstrates • An understanding of the methods' advantages and disadvantages • How to use the methods

Lovitts (2005) Component 2: Literature Review

The review • Is comprehensive and up to date • Shows a command/understanding of the literature • Contextualizes the problem • Includes a discussion of the literature that is selective, synthetic, analytical, and thematic Must use old text also, they could be critically important. For the most part we are interested in newer materials as well.

Lovitts (2005) Component 3: Theory

The theory that is applied or developed • Is appropriate • Is logically interpreted • Is well understood • Aligns with the question at hand • Shows comprehension of the theory's strengths and limitations

Bias

There are a number of reasons we should describe our participants (or the cases we actually analyze), rather than a sample: - Non-response or volunteer bias - Non-random or convenience sampling, "damaged" random sampling (randomness doesn't always work) - Transferability (a perhaps useful qualitative idea) - "Tell the story" of our participants • Quantitative Case Study

Variance

We will calculate using the formula: sum of squared deviations / N-1

z-score

a measure of how many standard deviations you are away from the norm (average or mean) Z scores that are larger are farther from the mean

Sample

a subset of the population

central tendency

add then divide by how many there are, you will find the mean

percentile rank

based on a large population, would assume a certain % of population would be at that score or below

Deviation

data - mean positive and negative mean how far it is from the mean Cannot just take the average of the deviation, we will get 0 because the mean is exactly centered.

Univariate

data that describes a single characteristic of the population Trying to describe the distribution of the data when we have scaled data or we are trying to look at frequencies and counts for data that are nominal and ordinal

Platokurtic

flat curve distribution

Population

group of individuals of the same species that live in the same area

Kurtosis

how flat or peaked a normal distribution is How tall is the data in the middle as compared to the other data

cumulative frequency cumulative proportion

how many people had that score or below what percentage was there or below

Deviation scores

mean will always be 0 SD will be the same as SD of scores Min and Max will be a little different

Leptokurtic

normal curves that are tall and thin, with only a few scores in the middle of the distribution having a high frequency Really tall, straight up and straight down

Unimodal

one peak, one modal interval

univariate, bivariate, multivariate

one variable, two variables, multiple variables

mu

represents the mean in the population you can use n or N to represent total people or N sub 1 to represent group 1.

x bar s

sample mean standard deviation of sample

Mean Absolute Deviation (Mean AD)

the average distance between each data value and the mean Convert each statistic into a positive value and divide by how many there are (absolute value). Add each absolute value and then divide by how many there are and that is the MAD You can also square every value instead of doing the absolute value. Add it up and that is the sum of sqaures. If it is divided by the total number that is population variance (data for everyone).

relative frequency

the proportion (or percent) of observations within a category. So percent of how many people had that score

Median

the score associated with the middle/median rank If you have 18 cases, 18+1 is 19/2 is 9.5, average rank 9 and 10s scores.

standard deviation

the square root of the variance. Interpreted as average distance from the mean (not exactly though). 2 SD is a cut point for smaller datasets 3 SD is a cut point for larger datasets

Standardized scale

z-scale. z-scores all on the same scale

population standard deviation

σ (sigma)

Evidence of Construct Validity

• "Unified" Construct Validity - Content (items) (do you have the right items to say what you are measuring) - Consequential (purposes, risks) (Are you using the instrument for the purpose it is intended, what is the risk? For example getting rid of ACT because of the risks it causes for students who do not do well, their bias) - Substantive (theoretical) (does it match the theory, the theory support it?) - Structural (statistical... factor analysis) (most statistical) - External (convergent-things related, discriminant, predictive) - Generalizability (settings, groups, tasks) (can we use the instrument across different settings, languages, populations)

But here are some things to avoid...

• A statistician is a professional who diligently collects facts and data and then carefully draws confusions about them. • Statisticians use data the way a fatigued man uses lamp-posts, for support rather than illumination.

Adapting Existing Instruments

• Adaptation of existing instruments - Get permission - Previous validity evidence may no longer apply - Impact of changing item wording • Change "leader" to "supervisor" - Adding/removing items is not trivial, must be tested - Cultural adaptation now preferred to simple translation or back -translation and is not trivial - Need new validity study process for changes

- Research problem

• An issue that must be addressed, solved

Brief overview of Internal Validity

• Are elements of a causal argument controlled in the design: (a) relationship, (b) temporal order (cause before effect), (c) control alternative explanations and confounding variables (done through research design, to control variables need to include them somehow), (d) replication (if something doesn't happen 100x scientists do not believe it happened), (e) theoretical understanding • What conclusions are legitimate based on your design? - All designs can be flawed, all designs can be useful. (Do not typically throw away entire study) • Internal validity & limitations • Design Validity (Brooks, 2011)

Research is a Rhetorical Process

• Because cause requires argument, I say that research is a rhetorical process - "of, relating to, or concerned with the art of speaking or writing formally and effectively especially as a way to persuade or influence people" (http://www.merriam-webster.com/dictionary/) • And I mean "rhetorical" in the best sense of the word, not the negative so often attributed to it • We need to convince ourselves and our audience we have found something useful

Try to Answer the Question First

• Before planning to do research, attempt to answer the research question using a literature review - If the research question has been answered, then research can focus on something else, perhaps still strongly related to the original question - If part of the research question has been answered, the research can be refined and focused to address the remaining unknowns - Need to be knowledgeable in the field/literature

Representative Validity

• Collect thoughtful and meaningful demographic characteristics for your participants - Gender, socioeconomic status, ethnicity - Perhaps other studied variables as well • Collect information that can be compared to known characteristics of the population • Useful trait and experiential variables that can be used to describe sample well People can read the description and decide how well you represented your sample. Convince reader you have a good sample

Types of Research Questions

• Decide/describe what type of research it will be: - Basic, applied (maybe not evaluation or action) - Safety, efficacy, or effectiveness research - Descriptive, exploratory, or confirmatory - Quantitative, qualitative, or mixed methods - Primary or secondary data analysis, or simulation - Replication or extension research - Prospective or retrospective - Longitudinal or cross-sectional

Descriptive Statistics

• Descriptive statistics help us understand the data before running statistical tests - and help us understand the statistical tests - For example, measurement statistics to verify that we can even run inferential statistics on these data in a meaningful way - Understanding the underlying subpopulations - Testing assumptions • But we can also ask significant, informative, descriptive Research Questions

Threats to Internal Validity

• Different design elements help us control threats to internal validity differently (better) • Pay attention to Threats to Internal Validity - Maturation - History - Testing (Priming) - Instrumentation - Selection Bias - Attrition (Mortality) - Statistical Regression - Contamination - Compensatory Rivalry - Experimenter bias - Resentful Demoralization - Selection-interactions Allow for both groups to mature at the same time, but do something different with one group Try to control threats with our designs

Brief overview of Theoretical Validity

• Does the research make sense? • Are there reasons to connect the variables? • Are there reasons to study the population? (why study this group?) • Why is the research important? "So what?" • In the proposal... and in the conclusions • Significance of the Study • Research is about theory not data • Theoretical Validity (Brooks, 2011)

Evidence of Construct Validity Explained

• Evidence of Construct Validity - Convergent/Discriminant validity - Predictive validity (using scales for hiring purposes, would like to have evidence that the scales work how they advertise. Like scales vs. one item measures for example course evaluation we want more questions not just one) - Multi-trait, Multi-method Matrix - Multiple measures of the same construct - Scales (vs. single-item measures) • Complex phenomena, Unreliability - Disparate populations (known-group differences) - Experimental Intervention (effective treatment)

Research topic

• General field of study, background to the study

External Validity... ... Evidence or Argument Needed Key arguments

• How Participants -Represent the Sample • How Sample -Represents the Sampling Frame • How Sampling Frame (or Accessible Population) -Represents the Target Population (no easy way to show that the subjects easily represents the target population. Not always true that people who choose to participate are similar to the people who have not chosen to participate) • How Target Population -Is well-defined and interesting

Sampling Process

• How did you select sample & how many? (large samples can represent the population, a good, small sample is often better) • When did you stop collecting data? • What's your response rate (sampling error) • How did you recruit participants? - Incentives, extra credit for class, employer (bias?) • How & How many times were they contacted? Who contacted them or collected data? Why? - Internet, Research assistant, Intermediary - Possibility an intermediary restricted participation?

Brief overview of External Validity

• How many, sampled how, from where? • The sample needs to represent the population • Do the cases have the information you need to answer your questions? • Is the case under study interesting or useful? • Generalizability and/or Transferability (do your results transfer from one study to another) • External validity & delimitations (how have you bounded your study, who is included) -Another way to think of this: Representative Validity (Brooks, 2011) If we randomly sample from our university, that is a convenient population, is it interesting or useful? Will it transfer?

External Validity

• How well do the participants or cases represent the population you are studying? (population external validity) • How well does the setting of the research represent the environment of the population? (ecological external validity) • How well does the timing of the study represent the time desired? (temporal external validity) (interesting to think about, data may differ in the beginning of the school year vs. the end of it)

Sampling Frame (also called Accessible Population)

• How well does your Accessible Population (or Sampling Frame) represent Target Population? • You need an actual list of population members in order to ensure an equal probability of all possible samples (random sampling) - Describe where and how you obtained your list • Internet search, organization, membership list -Think like lottery, need to be in the fishbowl to win. Some people at the party did not put in their name, they were part of the population, but not the target. The fishbowl is the accessible population • Can you actually access the Accessible Population? (e.g., gaining entry)

Sample (vs. Participants)

• How well does your Sample represent the Accessible Population? (coverage error) • Ideally, an equal probability random sampling method was used to create the sample • Note, the researcher decides whom to include in the sample - that is, the sample comprises those cases from a population that the researcher will request to participate in the study - BUT... not everyone agrees to participate (hence, "participants")

Brief overview of Statistical Conclusion Validity

• Impact of violations of assumptions, extreme values, range restriction, fishing (look for what we can find, maximize type 1 error), (dis)aggregation (Simpson's paradox), ..., on results? • What impact do potential errors (e.g., Type I, II) have on conclusions? • How large are the effects or relationships? • Be true to your data AND design • Statistical conclusion validity

"How to Grade a Dissertation" (Lovitts, 2005)

• In 2003-04, 272 faculty members at 9 universities, in 74 departments across 10 disciplines (science, social, humanities). On average, they were professors for 22 years, advised 13 dissertations, and served on 36 committees • Focus groups were asked to characterize dissertations and 6 components at 4 quality levels (outstanding to unacceptable) • Faculty members said they often make holistic judgments about the quality - no mental checklist of items against which they assess a dissertation - but results show that faculty members do make quality judgments that can be made explicit • By the way, she hated the title they gave her article She tried to find what faculty look for eve though they say they don't look for anything.

Proposal

• In journal articles, the introduction and literature review tend to be combined and are much shorter than in dissertations and theses • In academic writing, the proposal is typically designed for the student to assure the advisor and committee they can do the work Often, the proposal is seen as a '"contract" between the student and the faculty about what work will be done • This requires that strong effort be put forth in creating the proposal, so that it covers all the expected requirements for the research

Relationships among Variables

• Include support for why you are studying your variables and including other relevant variables (perhaps for statistical control) • Include support for why you are studying particular relationships among your variables (or differences among populations) • Include support for why you are excluding variables from your study (practical or theoretical reasons) - Why you don't need to include them

Don't Over-Reach

• Interpret analyses correctly - Don't over-interpret or go farther down the causal conclusion path than your design will allow - Report unusual analyses or tactics used (e.g., trim) - Connect conclusions to results, do not go beyond what the data will allow (e.g., extrapolation) - In external validity we USE random sampling as best chance for representativeness... • But in statistics we NEED random sampling for p values to be useful

Common Pitfalls in Research Design & Writing

• Note that there are entire separate courses dedicated to the topics addressed by these "validity" terms (and often several, like introductory and advanced) - Design - Measurement - Analysis • We can't possibly cover it all... but we'll do what we can

"Convenient Populations"

• Often our problem is not with the sample but with the accessible population we are studying • Instead of an Accessible Population that represents the Target Population well, some choose a Convenient Population • My suggestion for local generalizability is to draw a truly representative sample from that convenient population

Run Correct Analyses

• Perform the correct analyses • Avoid Type VI errors (analysis answers wrong question) • Interpret p values correctly if you use them • Pay attention to potential Null Hypothesis Testing errors • Always report appropriate effect sizes and/or confidence intervals • Remember that we can "buy" statistical significance • Consider carefully appropriate "post hoc" tests • Multiple hypothesis testing & data "fishing" • Many regression coefficients (especially dummy variables)

Detailed Procedures

• Procedures should be clear and transparent enough that if someone else needed to it, they would know how (like a recipe for research) (comes up a lot that people cannot reproduce results because the scientist did not explain all steps) • Said another way, other researchers need to be able to perfectly replicate your research • Defend your decisions to choose the methods and design, but don't defend the design itself - Qualitative researchers no longer need to explain why it's acceptable to do qualitative research

Target Population

• Provide delimitations of your Target Population - To whom will your results generalize? • Create strict Inclusion criteria - Who is in your target population • Create strict Exclusion criteria - Who is not, should not be in our screening • Use screening techniques or questions to include and exclude cases

Reliability (stability. But doesn't always work out because consistency may not be there)

• Reliability is a necessary but not sufficient requirement for Construct Validity • Evidence of Reliability - Test-retest - Alternate forms - Internal consistency - Inter-rater reliability - Inter-coder agreement - Item analysis

On the Emergence of Research Questions

• Research questions should generally not include the word "should" • "Should" questions are best answered by policymakers or decision-makers after they review the relevant research - Researchers rarely make decisions for others - Researchers want to remain objective rather than advocate for political positions • Need to be knowledgeable in the field/literature • Want broad experience in related fields • Use creativity techniques to think about possible questions (e.g., brainstorming) • Don't ask questions based only on the analytical methods you know how to use - We tend to ask questions we know how to ask - We tend to ask questions we know how to answer Once you've decided on your final research question, clearly write the question so that it can be answered empirically (using data you will collect or data that already exists) • You want to learn something useful from your research no matter what answer the results give you - Not every answer will change the world, but every answer should be useful What problem needs an answer? • Where are the gaps/contradictions in our knowledge? ' - If focus on very narrow gaps consider replication/extension • What assumptions can be changed? • Can it be applied differently? Any new tech? - Is the world different now? Any new ideas? • Any new theories/measurements for old ideas? • Do we dare upset folks with controversy? • Can we adapt from other fields? • Can it be newly qualitative or quantitative?

Response Sets and Bias

• Response sets - Acquiescence (people like to say yes and agree, want them to be honest not what they think will be helpful) - Social Desirability (people like to look good, may not answer honestly) - Extreme responding (reverse coding, choosing 1 or 5. Reverse the sentences- should not be all 5s, may not have read it correctly) - Moderate responding (remove middle, choosing 3) - Sponsor Bias (what they think you want to hear) - Consistency (answer new questions based on previous answers, rather than each independently)

Run Analysis Correctly

• Run correct analyses correctly - Test and/or report appropriate assumptions - Check for outliers and influential cases - Test appropriate interactions and main effects - Use appropriate control techniques where needed - Analyze appropriate units of analysis • Ecological fallacy, aggregation or disaggregation - Handle missing values appropriately - Cross-validation and replication of results • Provide "really useful" tables and graphs

- Research question

• Specific question to be answered by the research

Brief overview of Construct Validity

• The degree to which an instrument measures what it purports to measure (can we interpret the numbers the way we are going to do it) • Do the instruments provide the data needed? • Do the operational definitions match the theoretical or conceptual definitions? • Do the numbers mean what you say they mean? • Construct or Measurement Validity • Psychometric validity & Reliability

Said in another way...

• Theoretical Validity (the right reasons) • Internal Validity (the right methods) • External Validity (the right sources) • Construct Validity (the right information) • Conclusion Validity (the right answers) • Theoretical Validity (the right conclusions) - Note that there are entire courses dedicated to these validity topics

Pilot Study

• To make sure manipulations and equipment will work as you expect - To make sure data can be collected as desired - To make sure assistants know how to perform their roles - To make sure you have estimated time and costs and effort reasonably well • Interpret results only for minor, specific reasons • For pilot of scale, we recommend sample size of approximately n=30 (Johanson & Brooks, 2010)

Finagle's Rule

• To study a subject best, understand it thoroughly before you start - Know the literature well because it informs every aspect of the research process: research question, literature review, methods, results, and conclusions - Know everything about your research topic except the answer to the research question you are asking (Murphy's Law "Whatever can go wrong, will go wrong")

Writing about Research

• Traditionally 5 sections in 2 parts - To assure your committee you can do it ("contract") • The Proposal starts with these sections: - Introduction to the Research Problem - Literature Review (combined with intro in articles) - Research Design & Methods • Final Dissertation adds these sections: - Analysis & Results - Conclusions & Discussion

Basic Designs

• Treatment (X) then Observation (O) • X O • Observation then Treatment then Observation • O1 X O2 • Treatment then Observation with Control (i.e., nothing), Placebo, or Comparison Group (may have math majors in one class and then non-math majors in another class. Selection bias protentional) • X O • C O • No Treatment but Trait/Extant Group Comparisons (cannot be sure the group membership that caused those scores) • A O • B O

Existing Instruments

• Use of existing instruments - Get permission (some people had to retract articles because no permission) - Did they do validity studies (not just pilot studies, need to show it is useful) - You will "borrow" their validity evidence to justify your use of the instrument (i.e., provide evidence of previous use of the scale as part of your argument to use the existing instrument) - But you also need to collect your own evidence to verify that your own data are worth analyzing • Validity and reliability of scores, not scales FitBit example: every morning with shower get 7000 steps, its reliable because it happens everytime, but validity may not be correct because there is no steps taken

"Local Generalizability"

• We can rarely hope to truly represent a large target population, so let's represent the smaller, accessible convenient population well instead of not representing a population at all with a convenient sample • We definitely don't want convenient samples from convenient populations

Don't Need to Study it All at Once

• We don't necessarily need to do it all at once--- we can break the problem into multiple studies • If we are looking for relationships between variables A, B, and C - We can look at A, B, and C all in the same study - Or we can look at just A and B, or just B and C, or just A and C - If depends on what we specifically want to see now and how complicated a model we want to investigate (and analyze)

Construct Validity

• What are you measuring? (e.g., attitudes, perceptions, behaviors, practices, abilities) - Providing validity evidence that two attitude scales are correlated may be helpful, but it won't necessarily provide evidence that the attitude scale measures behaviors or abilities - Attitude predictive of behavior - Self-report data is always suspect - Do they ask a subordinate to respond when asking about leadership ability - What's the best source? (e.g., self- vs. other-report)

"Setting" in Quantitative Research

• What impact of the timing of data collection? - Maybe don't collect data immediately after proposal - Longitudinal data collection - Do seasons or times of year matter? - Does time of day matter? • Was your setting natural or artificial? - How realistic was the laboratory setting? • How much attrition occurred? • Do you need additional permissions for access? Southeast Ohio internet vs. Columbus internet

- Statement of the Purpose

• What you plan to study within that field, related to that problem

Why Research?

• Why do we do research? - Develop, improve, confirm, or disconfirm theory - Rigorous attempt to answer questions - Share our answers with others • Our instinctual reaction in answer to a question is not always correct - Will 2 balls hit the ground at the same time even if one weighs more than the other? - Will putting a Mentos in Diet Coke cause explosion?

Better Designs

• with Control Group and Pretest ("quasi") (pre-test to see if there is improvement in the end) • O1 X O2 • O1 C O2 • Random Assignment before Treatment then Observation with Control Group • R X O • R C O • Random Assignment before Observation then Treatment then Observation with Control Group • R O1 X O2 • R O1 C O Random assignment tries to create equivalent groups across all variables