PSY 3001W EXAM #2
Sampling frame:
Actual population from which samples will be drawn
Properties of measurement scales:
1) Magnitude 2) Equal intervals 3) Absolute zero
Ways to test reliability:
1) Test-retest reliability 2) Parallel forms reliability 3) Internal consistency reliability 4) Interrater/interobserver reliability
2 pieces of information we can get from r:
1) The sign of r tells us whether there is a positive or negative relationship 2) r^2 tells us the proportion of variance in Y that is attributable to X (strength of the relationship)
Two components of a measurement:
1) True score 2) Measurement error Poorly worded questions; unclear instructions; inconsistent measurement conditions; mistakes/inaccuracies by the measurer
6 Steps in constructing a questionnaire:
1. Decide what information you are looking for 2. Choose a format 3. Write a first draft 4. Reexamine and revise the questionnaire 5. Pretest the questionnaire 6. Edit the questionnaire and specify guidelines for its use
Unobtrusive measures may or may not...
Involve intervention by the researcher\ • Evaluate fingerprint smudges on book pages to evaluate usage
Validity:
Is the raw measurement a good operational definition of some construct?
Negative slope:
Negative or inverse relationship
Double-blind experiment:
Neither experimenter nor participants know what conditions participants are assigned to
Reliability:
Does the measurement instrument give us the same answer every time?
Linear regression:
Finding the best-fitting line regression coefficients: Slope and intercept • Y= mx+b • m= slope • b= intercept
Amount of measurement error is related to...
The variance of the measurements
Obtain demographic information for possible later use
e.g. male/female, race, education level
Check for attention or...
Random responding
Multistage Sampling:
Randomly select large clusters, then randomly select smaller clusters within the larger clusters
Simple random sampling:
Randomly select participants from the population of interest • Does not guarantee a representative sample
Invasive measures:
Recording activity of individual neurons with micro electrodes • Often, when using physiological measures, you must infer psychological states from physiological states
Snowball sampling:
Recruit some participants and ask them to identify other potential participants
Participation:
Researcher becomes a member of the group that is under observation • Problem: being a participant can affect objectivity, observer might influence behavior of others
Situation Sampling:
Sample behaviors in many different situations
Representative sample:
Sample characteristics closely match population characteristics
r is a...
Sample statistic
Haphazard sampling (Convenience sampling):
Select whomever happens to come along • May limit generalizability (external validity)
Sampling:
Selecting a sample to draw conclusions about the population
Floor effect:
Task is so difficult that no one performs well
Ceiling effect:
Task is so easy that everyone performs well
Double-barreled items:
The furnishings in the waiting room are comfortable, and there is plenty of interesting reading material instead of When I call for an appointment, the appointment clerk answers promptly and is courteous and efficient
Pilot Study:
• "Trail run" for experiment • Shows potential for success • Reveals problems in design
Sensitivity of the DV:
• A rating scale can reveal smaller differences than a yes-no scale
Test-retest reliability:
• Administer the same test at two different times • Compute correlation between 1st and 2nd scores • r > .80 considered good reliability
Measuring the DV: Types of Measures
• Behavioral measures • Physiological measures • Self report Measures
Probability:
• Simple random sampling • Stratified random sampling • Cluster sampling • Multistage sampling
Guidelines for writing survey items:
• Simplicity • Avoid double-barreled items • Avoid leading items • Avoid loaded items • Avoid negative wording, especially double negatives • Phrase questions to detect "yea saying" and "nay-saying"
Internal consistency reliability:
• Split-half reliability • Compute scores for two halves of the test • Compute correlation between scores on two halves of test
Archival Research:
• Statistical Records • Written and Mass Communication Records
Problems of test-restest reliability:
• Subjects may remember how they responded the first time • Variable being measured may change over time
Parallel forms reliability:
• Tests the equivalence between two different forms of the same test (e.g. memory tests) • Administer both forms to many people • Compute correlation between scores on the two tests
Solutions to expectancy problems:
• Train experimenters • Print out instructions to participants • Automate procedures • Use experimenters who are "blind" to IV manipulation
Effects due to participant expectations:
• Use a placebo control group to control for expected effects of a treatment • Use deception or withhold some details of experiment to keep participants from figuring out topic of interest • Ask participant questions during debriefing
How will you obtain participants?
• Use convenience sampling (recruit whoever is available) • Convenience sampling may affect your ability to generalize your results to other populations • Usually generalization is not a problem because behavior often is not specific to a particular population
Cluster sampling:
• Useful in situations when you don't have a list of all members of the population • Identify naturally occurring groups (clusters) • Randomly select clusters • Survey all participants in the selected clusters • Easy and quick, but people in the selected clusters may not be like those in other clusters
Reliability:
•Repeatability or consistency of a measurement • Extent to which the measurement refelcts the true value of what you are measuring
Construct validity:
•The extent to which a measure measures what it's supposed to measure • Face validity • Must be operationally defined
Experimenter Bias/Expectancy effects:
•When experimenter expects particular behavior and acts in a way to cause that behavior to occur • Clever Hans • Expectancy effects can occur when experimenter knows what groups participants are in and treats them differently e.g., rats described as "bright" or "dull" learns mazes consistent with labels Must be due to differential treatment by trainers
In multiple choice exams...
make sure options are mutually exclusive and exhaustive
Carefully define points on rating scales...
particularly end-poitnts
Facilitated communication is an example of...
Expectation effects
Concurrent validity:
Give a test and then immediately evaluate criterion behavior EX: pencil-and-paper test of navigation knowledge, then immediately observe and rate person's behavior in flight
Predictive validity:
Give a test at some point in time, comapre score to future criterion behavior EX: GRE score and grad school performance; test of coordination/attention and productivity on an assembly line
Measures of variability:
Variance, standard deviation, range
Questionnaires vs. Interviews:
What is the difference?
Measurement scale determines....
What statistical analyses can be performed
Other factors that affect the quality (representativeness) of samples:
• Sampling frame • Response rate
Reliability is not the same as...
"Correctness"
1st category of measurement scale:
1. Ratio Magnitude, equal intervals, absolute zero ex: weight, salary In a ratio scale, ratios have meaning (twice as much money)
2nd category of measurement scale:
2. Interval Magnitude, equal intervals No absolute zero ex: temperature in degrees Fahrenheit 40 degrees is not twice as much as 20 degrees A particular difference is the same size anywhere on the scale
3rd category of measurement scale:
3. Ordinal Magnitude only ex: rankings Difference between 1st and 2nd may be different than from 2nd to 3rd Only tells you relative standing, not distance between values
4th category of measurement scale:
4. Nominal None of the three properties Classification of items into groups that have no magnitude relationship ex: nationality, gender, category names Might involve numbers, but the numbers are just labels
Sampling:
An issue in systematic observation
Demand characteristics:
Any cues or information about the experiment that might guide participant behavior
Simplicity:
Appointments are arranged expeditously
Random Sampling:
Avoids biasing
Measurement:
Assigning numbers or labels to observations to represent amounts or categories
Choose a measurement scale that can...
Be generalized to real-world situations of interest
Content analysis:
Coding system
Naturalistic Observation is a form of...
Case study that focuses on a particular situation • Has the same shortcomings as a case study • Inability to rule out alternative explanations • Inability to say what would have happened in other circumstances
Case Study:
Detailed observation of a single person or situation, often to understand the development of some (usually uncommon) condition • Provide direction for future research • Limited usefulness beyond that • Cannot provide evidence to support one theory over another • Lack comparative information that allows us to rule out alternative explanations
Concealment:
Disguised vs. undisguised observation • Addresses problems with reactivity (when observation changes subject's behavior) • In undisguised observation, use desensitization and habituation
Use filter questions to avoid wasting respondent's time
Do you own a car? (If not, skip to question 47)
Interrater reliability:
Extent to which the observations of different observers agree • Raters are often trained extensively • Carefully developed coding system leads to precise descriptions
Nonreactive or unobtrusive measures...
Indirect observations of behavior EX: Observe graffiti • Count beer bottles in trash • Examine trash to assess food consumption
Loaded items:
Items that contain emotion-laden terms or that suggest a socially desirable response EX: I would be open-minded with respect to the use of radical new treatment plans instead use I would be open-minded with respect to the use of new treatment plans
Leading items:
Items that suggest a particular response
Measures of central tendency:
Mean, median, mode
An unreliable measure contains a large proportion of...
Measurement error
Manipulation Check:
Measures whether IV manipulation had intended effect on participants
Disadvantage of naturalistic observation:
No control by the researcher, therefore no inference of cause and effect (at best, correlational research)
Naturalistic Observation:
Observation of behavior without any attempt to intervene • Advantages: external validity, avoids some ethical problems • Description and Interpretation of Data: usually qualitative
Response rate:
Percentage of sample that responds • Low response rate may indicate a biased sample (response bias)
Positive slope:
Positive or direct relationship
Behavioral predictions are based on...
Probability • There will always be exceptions and outliers
Coding System:
Process of data reduction
Ask the same type of question multiple times to check...
Reliability
Measurement scale:
Set of possible numbers or labels that can be assigned to measured items
Purposive sampling:
Similar to haphazard sampling, except that you choose participants that satisfy certain criteria (age, gender, etc.)
Response bias:
Some individuals chosen to respond to a survey systematically fail to do so
Systematic Observation:
Study of one or more specific behaviors in a particular setting • Typically begins with a narrative record • Then the narrative record is analyzed for occurrences of specific behaviors of interest
Reactivity:
Subject's behavior changes because of awareness of observer • Allow subject to get used to observer adn environment • Conceal observer • Allow for anonymous responding
p (rho) is...
The corresponding population parameter
Negative wording:
The doctors do not order unnecessary tests, nor do they fail to order necessary ones
Advantage of naturalistic observation:
Useful for describing behavior and relationships between variables under natural circumstances (external validity)
Spurious relationship:
When the relationship between two variables can be attributed to a third variable (or chance) ex: Positive correlation between a city's ice cream sales and drownings in city pools Both are related to outdoor temperature
Observing behavior: Quantitative Approaches
• Choose behavior of interest (helping behavior, approach/avoidance, behavior, eye contact) • Choose measure of behavior (eye contact: yes or no; duration of eye contact) • Allows us to perform statistical analyses
Quota sampling:
• Combination of stratified random sampling and haphazard sampling • Identify strata that are relevant to your topic • Select appropriate number from each strata to yield same proportions as in population
Stratified random sampling:
• Divide population into strata • Select a random sample (often of equal size) from each stratum • Guarantees that some of each group will be in sample • Might lead to overrepresentation of a group
Physiological measures:
• EEG • PET • GSR • MRI
Manipulating the IV: Staged manipulation
• Events are staged to manipulate the IV, often using confederates (accomplices) • Select levels of IV to achieve largest effect
Open-ended questions:
• Flexibility in responding • Difficult to summarize; often requires a coding scheme
Non-probability sampling:
• Haphazard sampling • Purposive sampling • Quota sampling • Snowball sampling
Questionnaires:
• Inexpensive • Difficult to motivate respondents • Neither respondent nor researcher can seek clarification • Response rate can be low
Research Proposal:
• Like introduction and methods sections of a manuscript • Often peer-reviewed; provides opportunity to receive feedback • Gives organization to proposed experiements
Interviews:
• More expensive to administer • Higher response rate, less chance of response bias • Respondent can ask questions, interviewer can ask for clarification of responses • Possibility of interview bias, when the interviewer inadvertently biases responses by reacting to responses or guiding responses
Closed questions:
• Multiple choice • Yes/No • Rating scale • Easier to summarize • Reduces expressiveness and spontaneity
Interrater/Interobserver reliability:
• Multiple observers rate many observations • Compute correlation between ratings
Qualitative Approaches:
• Narrative record • Written, audio, or video • Behaviors are then classified and organized • Record should be made as soon as possible
Choosing a scale of measurement:
• Nominal: difference in quality • Ordinal: crude information about quantity • Interval: also tells us how much scores differ • Ratio: tells us how much of the quality is/was present
When r = + or - 1.0:
• Only a portion of the variability in one variable is related to variability in the other • Predictions of one variable from the other will be imperfect • Other factors influence the two variables (e.g. error variance)
Survey formats:
• Open-ended questions • Close questions
Self-Report measures:
• Participant might report using a rating scale or yes/no response • Self report measures have potential reliability and validity problems EX: • participant may not be aware of true thought processes • participant may not recall accurately
Correct way to test theory of subliminal perception with respect to the gin ad:
• Present a gin ad with messages to one group of people and a gin ad with no messages to another • If 62% in the "message" group report arousal and only 10% in the "no message" group report arousal, conclude the message had an effect • If 62% in both groups report arousal, conclude the messages had no effect
Sampling techniques:
• Probability • Non-probability
Reporting results of research:
• Professional meetings • Peer-reviewed journals
Manipulating the IV: Setting the Stage
• Provide informed consent info • Describe purpose of study
Behavioral measures:
• Record whether or not participant responds • Record the frequency of behaviors • Measure reaction time (latency) • Measure duration of response • Count number of errors or number correct