PSY 3001W EXAM #2

Sampling frame:

Actual population from which samples will be drawn

Properties of measurement scales:

1) Magnitude 2) Equal intervals 3) Absolute zero

Ways to test reliability:

1) Test-retest reliability 2) Parallel forms reliability 3) Internal consistency reliability 4) Interrater/interobserver reliability

2 pieces of information we can get from r:

1) The sign of r tells us whether there is a positive or negative relationship 2) r^2 tells us the proportion of variance in Y that is attributable to X (strength of the relationship)

Two components of a measurement:

1) True score 2) Measurement error Poorly worded questions; unclear instructions; inconsistent measurement conditions; mistakes/inaccuracies by the measurer

6 Steps in constructing a questionnaire:

1. Decide what information you are looking for 2. Choose a format 3. Write a first draft 4. Reexamine and revise the questionnaire 5. Pretest the questionnaire 6. Edit the questionnaire and specify guidelines for its use

Unobtrusive measures may or may not...

Involve intervention by the researcher\ • Evaluate fingerprint smudges on book pages to evaluate usage

Validity:

Is the raw measurement a good operational definition of some construct?

Negative slope:

Negative or inverse relationship

Double-blind experiment:

Neither experimenter nor participants know what conditions participants are assigned to

Reliability:

Does the measurement instrument give us the same answer every time?

Linear regression:

Finding the best-fitting line regression coefficients: Slope and intercept • Y= mx+b • m= slope • b= intercept

Amount of measurement error is related to...

The variance of the measurements

Obtain demographic information for possible later use

e.g. male/female, race, education level

Check for attention or...

Random responding

Multistage Sampling:

Randomly select large clusters, then randomly select smaller clusters within the larger clusters

Simple random sampling:

Randomly select participants from the population of interest • Does not guarantee a representative sample

Invasive measures:

Recording activity of individual neurons with micro electrodes • Often, when using physiological measures, you must infer psychological states from physiological states

Snowball sampling:

Recruit some participants and ask them to identify other potential participants

Participation:

Researcher becomes a member of the group that is under observation • Problem: being a participant can affect objectivity, observer might influence behavior of others

Situation Sampling:

Sample behaviors in many different situations

Representative sample:

Sample characteristics closely match population characteristics

r is a...

Sample statistic

Haphazard sampling (Convenience sampling):

Select whomever happens to come along • May limit generalizability (external validity)

Sampling:

Selecting a sample to draw conclusions about the population

Floor effect:

Task is so difficult that no one performs well

Ceiling effect:

Task is so easy that everyone performs well

Double-barreled items:

The furnishings in the waiting room are comfortable, and there is plenty of interesting reading material instead of When I call for an appointment, the appointment clerk answers promptly and is courteous and efficient

Pilot Study:

• "Trail run" for experiment • Shows potential for success • Reveals problems in design

Sensitivity of the DV:

• A rating scale can reveal smaller differences than a yes-no scale

Test-retest reliability:

• Administer the same test at two different times • Compute correlation between 1st and 2nd scores • r > .80 considered good reliability

Measuring the DV: Types of Measures

• Behavioral measures • Physiological measures • Self report Measures

Probability:

• Simple random sampling • Stratified random sampling • Cluster sampling • Multistage sampling

Guidelines for writing survey items:

• Simplicity • Avoid double-barreled items • Avoid leading items • Avoid loaded items • Avoid negative wording, especially double negatives • Phrase questions to detect "yea saying" and "nay-saying"

Internal consistency reliability:

• Split-half reliability • Compute scores for two halves of the test • Compute correlation between scores on two halves of test

Archival Research:

• Statistical Records • Written and Mass Communication Records

Problems of test-restest reliability:

• Subjects may remember how they responded the first time • Variable being measured may change over time

Parallel forms reliability:

• Tests the equivalence between two different forms of the same test (e.g. memory tests) • Administer both forms to many people • Compute correlation between scores on the two tests

Solutions to expectancy problems:

• Train experimenters • Print out instructions to participants • Automate procedures • Use experimenters who are "blind" to IV manipulation

Effects due to participant expectations:

• Use a placebo control group to control for expected effects of a treatment • Use deception or withhold some details of experiment to keep participants from figuring out topic of interest • Ask participant questions during debriefing

How will you obtain participants?

• Use convenience sampling (recruit whoever is available) • Convenience sampling may affect your ability to generalize your results to other populations • Usually generalization is not a problem because behavior often is not specific to a particular population

Cluster sampling:

• Useful in situations when you don't have a list of all members of the population • Identify naturally occurring groups (clusters) • Randomly select clusters • Survey all participants in the selected clusters • Easy and quick, but people in the selected clusters may not be like those in other clusters

Reliability:

•Repeatability or consistency of a measurement • Extent to which the measurement refelcts the true value of what you are measuring

Construct validity:

•The extent to which a measure measures what it's supposed to measure • Face validity • Must be operationally defined

Experimenter Bias/Expectancy effects:

•When experimenter expects particular behavior and acts in a way to cause that behavior to occur • Clever Hans • Expectancy effects can occur when experimenter knows what groups participants are in and treats them differently e.g., rats described as "bright" or "dull" learns mazes consistent with labels Must be due to differential treatment by trainers

In multiple choice exams...

make sure options are mutually exclusive and exhaustive

Carefully define points on rating scales...

particularly end-poitnts

Facilitated communication is an example of...

Expectation effects

Concurrent validity:

Give a test and then immediately evaluate criterion behavior EX: pencil-and-paper test of navigation knowledge, then immediately observe and rate person's behavior in flight

Predictive validity:

Give a test at some point in time, comapre score to future criterion behavior EX: GRE score and grad school performance; test of coordination/attention and productivity on an assembly line

Measures of variability:

Variance, standard deviation, range

Questionnaires vs. Interviews:

What is the difference?

Measurement scale determines....

What statistical analyses can be performed

Other factors that affect the quality (representativeness) of samples:

• Sampling frame • Response rate

Reliability is not the same as...

"Correctness"

1st category of measurement scale:

1. Ratio Magnitude, equal intervals, absolute zero ex: weight, salary In a ratio scale, ratios have meaning (twice as much money)

2nd category of measurement scale:

2. Interval Magnitude, equal intervals No absolute zero ex: temperature in degrees Fahrenheit 40 degrees is not twice as much as 20 degrees A particular difference is the same size anywhere on the scale

3rd category of measurement scale:

3. Ordinal Magnitude only ex: rankings Difference between 1st and 2nd may be different than from 2nd to 3rd Only tells you relative standing, not distance between values

4th category of measurement scale:

4. Nominal None of the three properties Classification of items into groups that have no magnitude relationship ex: nationality, gender, category names Might involve numbers, but the numbers are just labels

Sampling:

An issue in systematic observation

Demand characteristics:

Any cues or information about the experiment that might guide participant behavior

Simplicity:

Appointments are arranged expeditously

Random Sampling:

Avoids biasing

Measurement:

Assigning numbers or labels to observations to represent amounts or categories

Choose a measurement scale that can...

Be generalized to real-world situations of interest

Content analysis:

Coding system

Naturalistic Observation is a form of...

Case study that focuses on a particular situation • Has the same shortcomings as a case study • Inability to rule out alternative explanations • Inability to say what would have happened in other circumstances

Case Study:

Detailed observation of a single person or situation, often to understand the development of some (usually uncommon) condition • Provide direction for future research • Limited usefulness beyond that • Cannot provide evidence to support one theory over another • Lack comparative information that allows us to rule out alternative explanations

Concealment:

Disguised vs. undisguised observation • Addresses problems with reactivity (when observation changes subject's behavior) • In undisguised observation, use desensitization and habituation

Use filter questions to avoid wasting respondent's time

Do you own a car? (If not, skip to question 47)

Interrater reliability:

Extent to which the observations of different observers agree • Raters are often trained extensively • Carefully developed coding system leads to precise descriptions

Nonreactive or unobtrusive measures...

Indirect observations of behavior EX: Observe graffiti • Count beer bottles in trash • Examine trash to assess food consumption

Loaded items:

Items that contain emotion-laden terms or that suggest a socially desirable response EX: I would be open-minded with respect to the use of radical new treatment plans instead use I would be open-minded with respect to the use of new treatment plans

Leading items:

Items that suggest a particular response

Measures of central tendency:

Mean, median, mode

An unreliable measure contains a large proportion of...

Measurement error

Manipulation Check:

Measures whether IV manipulation had intended effect on participants

Disadvantage of naturalistic observation:

No control by the researcher, therefore no inference of cause and effect (at best, correlational research)

Naturalistic Observation:

Observation of behavior without any attempt to intervene • Advantages: external validity, avoids some ethical problems • Description and Interpretation of Data: usually qualitative

Response rate:

Percentage of sample that responds • Low response rate may indicate a biased sample (response bias)

Positive slope:

Positive or direct relationship

Behavioral predictions are based on...

Probability • There will always be exceptions and outliers

Coding System:

Process of data reduction

Ask the same type of question multiple times to check...

Reliability

Measurement scale:

Set of possible numbers or labels that can be assigned to measured items

Purposive sampling:

Similar to haphazard sampling, except that you choose participants that satisfy certain criteria (age, gender, etc.)

Response bias:

Some individuals chosen to respond to a survey systematically fail to do so

Systematic Observation:

Study of one or more specific behaviors in a particular setting • Typically begins with a narrative record • Then the narrative record is analyzed for occurrences of specific behaviors of interest

Reactivity:

Subject's behavior changes because of awareness of observer • Allow subject to get used to observer adn environment • Conceal observer • Allow for anonymous responding

p (rho) is...

The corresponding population parameter

Negative wording:

The doctors do not order unnecessary tests, nor do they fail to order necessary ones

Advantage of naturalistic observation:

Useful for describing behavior and relationships between variables under natural circumstances (external validity)

Spurious relationship:

When the relationship between two variables can be attributed to a third variable (or chance) ex: Positive correlation between a city's ice cream sales and drownings in city pools Both are related to outdoor temperature

Observing behavior: Quantitative Approaches

• Choose behavior of interest (helping behavior, approach/avoidance, behavior, eye contact) • Choose measure of behavior (eye contact: yes or no; duration of eye contact) • Allows us to perform statistical analyses

Quota sampling:

• Combination of stratified random sampling and haphazard sampling • Identify strata that are relevant to your topic • Select appropriate number from each strata to yield same proportions as in population

Stratified random sampling:

• Divide population into strata • Select a random sample (often of equal size) from each stratum • Guarantees that some of each group will be in sample • Might lead to overrepresentation of a group

Physiological measures:

• EEG • PET • GSR • MRI

Manipulating the IV: Staged manipulation

• Events are staged to manipulate the IV, often using confederates (accomplices) • Select levels of IV to achieve largest effect

Open-ended questions:

• Flexibility in responding • Difficult to summarize; often requires a coding scheme

Non-probability sampling:

• Haphazard sampling • Purposive sampling • Quota sampling • Snowball sampling

Questionnaires:

• Inexpensive • Difficult to motivate respondents • Neither respondent nor researcher can seek clarification • Response rate can be low

Research Proposal:

• Like introduction and methods sections of a manuscript • Often peer-reviewed; provides opportunity to receive feedback • Gives organization to proposed experiements

Interviews:

• More expensive to administer • Higher response rate, less chance of response bias • Respondent can ask questions, interviewer can ask for clarification of responses • Possibility of interview bias, when the interviewer inadvertently biases responses by reacting to responses or guiding responses

Closed questions:

• Multiple choice • Yes/No • Rating scale • Easier to summarize • Reduces expressiveness and spontaneity

Interrater/Interobserver reliability:

• Multiple observers rate many observations • Compute correlation between ratings

Qualitative Approaches:

• Narrative record • Written, audio, or video • Behaviors are then classified and organized • Record should be made as soon as possible

Choosing a scale of measurement:

• Nominal: difference in quality • Ordinal: crude information about quantity • Interval: also tells us how much scores differ • Ratio: tells us how much of the quality is/was present

When r = + or - 1.0:

• Only a portion of the variability in one variable is related to variability in the other • Predictions of one variable from the other will be imperfect • Other factors influence the two variables (e.g. error variance)

Survey formats:

• Open-ended questions • Close questions

Self-Report measures:

• Participant might report using a rating scale or yes/no response • Self report measures have potential reliability and validity problems EX: • participant may not be aware of true thought processes • participant may not recall accurately

Correct way to test theory of subliminal perception with respect to the gin ad:

• Present a gin ad with messages to one group of people and a gin ad with no messages to another • If 62% in the "message" group report arousal and only 10% in the "no message" group report arousal, conclude the message had an effect • If 62% in both groups report arousal, conclude the messages had no effect

Sampling techniques:

• Probability • Non-probability

Reporting results of research:

• Professional meetings • Peer-reviewed journals

Manipulating the IV: Setting the Stage

• Provide informed consent info • Describe purpose of study

Behavioral measures:

• Record whether or not participant responds • Record the frequency of behaviors • Measure reaction time (latency) • Measure duration of response • Count number of errors or number correct