Methods Test two
Extraneous Varaible
a variable that influences both the independent and dependent variables-creating a spurious association between them, which disappears when the extraneous variable is controlled
How do researchers try to verify some of the information in the FFQ when they know there might be a lot of error?
with validation studies that check FFQ results against those obtained via other methods, usually a 24-hour food recall/food diaries that is very long/over a couple of days
Confidentiality(privacy)
when no one except the researcher know the identities of the participants or information about them can be waived
Anonymity
when no one, including the researcher, knows the identities of the participant
Many omnibus surveys use a split-ballot design
when some questions are only asked of a sub-set of randomly selected respondents
contextual effect
when the context in which something occurs changes whether the effect happens (A changes B)
Spurious relationship/association
when there is no meaningful causal connection between apparently associated variables because the relationship is caused by a third variable
Causality(cause)
-"A" causes "B" to happen i.e. remove A and nothing will happen to B
Trend Study
- a study design in which data are collected at two or more points in time from a different sample of the same population -disadvantage: you're not talking to the same people so there is not strong confidence -advantages: useful when researchers are interested in identifying changes over time in a large population -avoids the time and expense of relocating the same individuals to track changes
Cross-Sectional Study Design
-A study design in which data are collected at only one point in time -most frequently used design in social science research -advantages: faster and cheaper -disadvantages: issue with causality, can't establish time order
Consider if you need:
-Filter questions: a question used to identify a subset of respondents who are then asked additional questions -Contingent questions: a question that is asked only of a subset of survey respondents, depending on their response to an earlier question examples: Do you have a job (how # hours, salary?) Have you ever been pregnant(when? how #?) Are you a college athlete(sport? how long?)
What can we do about these challenges?
-Find out as much information about the original authors of the data set and their methods as possible 1. what were the researcher's goals in collecting the data? 2. what data was collected, and what were they trying to measure? 3. when was the information collected? 4. what methods were used for data collection? 5. what is known about the success of the data-collection effort? How are missing data indicated? -use multiple indicators of key concepts -test for the stability of theoretical relationships(findings) across sample sub-groups
What are a few commonly used datasets?
-General Social Survey(GSS) -Cencus -Detroit Area Survey -ADD Health -Current Population Survey(CPS) -American Community Survey(ACS)
Cohort
-a group of people born within a given time frame or experiencing a life event, such as marriage or graduation from high school, in the same time period -examples: class of 2017, generation x, born 1994-1998, people who went to Vietnam
Longitudinal Designs
-a research design in which data are collected at least two different times -may be a panel, trend, or cohort study -disadvantages: very expensive -advantages: can accurately track changes over time, can establish time order(and therefore causality), doesn't rely on asking retrospective questions(as would a cross-sectional design)=high validity, can do both cross-sectional and over-time analyses on each time period, so allows for more analysis
Panel Study
-a study design in which data are collected at two or more points in time from the same individuals -advantages: best way to truly see change over time within people, allows for more confidence in causality -disadvantages: expensive, attrition can decrease validity
Moderator(Moderating Variable)
-a variable that identifies that context for the effect of other variables -example: in high-income communities, gender has no effect on likelihood of achieving a high-school diploma, but it does in all others
Uses of spilt ballot design
-allows more different questions to be asked within one "wave" of the sample -allows researchers to ask the same question two different ways and compare the results
Ecological Fallecy
-an error in reasoning when incorrect conclusions(i.e. causal claims) about individual-level processes are drawn from group-level data -sometimes your conclusions may be correct doing this, sometimes wrong(there is no proof-it's just a correlation!) -example: factories with more unskilled workers have more incidents of sabotage occur
Reductionist Fallecy
-an error in reasoning when incorrect individual-level conclusions are drawn about complex phenomenon with group-level factors -uses individual data-but doesn't recognize larger social factors like biological or psychological factors -example:blaming school shootings on the mental health of the shooter
Intervening Variables or Mediators
-are variables that occur between the independent and dependent variable and are involved in a causal mechanism -may be identified using quantitative or qualitative research
Mailed(self- administered) surveys
-biggest benefit? cost! cheaper than phone, or in person -biggest concern? Low-response rate! what is a target response rate in survey research? 70% response rate
Electronic surveys
-biggest benefits? can collect sensitive data that might be embarrassing to admit in person, can gather data very fast(within a few days) -biggest concerns? sampling error & response rates! not everyone has access to the internet-patterns in who does not. some people don't want to take internet surveys, need to get the survey to the right people(& only the right people) to be able to get a representative sample
Telephone surveys
-biggest benefits? easy to reach people quickly(can gather data very fast), cost=cheaper than in-person, traditionally higher response rates than by mail -biggest concerns? sampling issues: finding valid #s for proper sampling units, response rates:getting a hold of people and getting them to participate, distractions for respondents:they may be driving or talking to other people
Rules for Response Categories/Attributes
-choices need to be balanced -should be mutually exclusive and exhaustive -choices need to make sense in connection to the question
Event Based Design
-cohort studies that focus on people who are all experiencing the same event at the same time
What are the benefits of secondary data analysis?
-cost is very inexpensive -time: don't have to collect your own data; work of creating and testing measures has already been done -amount of data:allows for inclusions of more variables and larger, more diverse sample than other feasible -access: allow for analysis of social processes in locations that are inaccessible -comparisons:it can facilitate comparison with other samples and/or allow data from multiple studies to be combined(depending on methods)
National Ethics Commision
-due to these and other studies that violated human rights, the U.S. created the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research -produced the Belmont Report (1979), which was transformed into the Federal Policy for the Protection of Human Subjects(1991)
Surveys are the most popular form of social research. What are some of their advantages?
-efficient: less time/cost per person than interviews or participant observation/ethnography -generalizability: lend themselves to probability sampling from large populations -allows for sophisticated statistical analysis to be done:unfortunately(for survey researchers), new technologies are slightly lowering the overall rate of response to surveys and making it more difficult to obtain generalizable samples -when cross-population generalizability is a key concern, they allow a range of social contexts and subgroups to be sampled
Why is qualitative secondary data analysis more difficult?
-far fewer qualitative datasets are available -harder to fully understand the social context with only the transcripts and you can't ask follow up questions if you don't fully understand -can't work completely inductively -it is not possible to engage in the iterative process of making observations, developing concepts, making more observations, and refining the concepts, which is the hallmark of much qualitative methodology
How many response categories are usually desirable?
-five categories work well for unipolar ratings -seven will capture most variation on bipolar ratings
Is all deception bad? What do the rules say about deception?
-guidelines say to avoid deception in research, except in limited circumstances when it is unavoidable -violates principles of informed consent -research must fully "debrief" participant after to meet ethical obligation
Bipolar Rating
-include a middle category and parallel responses with positive and negative valence(can be labeled or unlabeled) -ex. Very comfortable, neither very or uncomfortable, very uncomfortable
Survey Research
-involves the collection of information from a sample of individuals through their responses to questions about themselves or others -may be done in person, over the phone or online -when done in person, may also be called quantitative interview or structured interview
Advantages and Disadvantages of Cohort Studies
-it depends on whether either panel or trend study design are being used(either may be used) -trend study designs are the "default" method, as they are more often done
Surveys (& interviews) rely upon self-reported data. What are some of the many reasons, why this might be inaccurate?
-may be lying to researchers -may be lying to themselves -may not understand the question -may not have any knowledge of the topic -may have inaccurate memories of the event -current mood might be influencing general attitudes -social desirability bias
Consent
-must be given by a person "competent" to consent -they must be consented voluntarily
What ethical issues are common with secondary data analysis?
-not very many -you don't even need to get approval from IRB because it doesn't involve any data collection from people -you should always evaluate if you think the original researchers didn't have confidentiality which can be at risk-more likely an issue for qualitative data
Fence-sitters
-people who see themselves as being neutral, may skew the results if you force them to choose sides -solution? need to provide a middle, neutral alternative within the response categories
What does randomization have to do with causality?
-randomization is used to reduce the risk of spuriousness -true experiments should all use randomization for assignment to the experimental and control group
Three basic ethical principles for the protection of human subjects were establushed
-respect for persons: treating persons as autonomous agents and protecting those with diminished autonomy -beneficence: minimizing possible harms and maximizing benefits -justice: distributing benefits and risks of research fairly
Floaters
-respondents who choose a substantive answer when they really don't know or have no opinion -solution? provide an "explicit" response category of "don't know" -because some people have opinions but are reluctant to give them, sometimes researchers will still choose not to do this(ex. political polling)
How?
-send a direct link to the survey website to each participant and require that they enter a PIN to access the survey -only works with a complete sampling frame- so not usually -message should be clear and interesting with a catchy subject line
What can we do to increase the response rate?
-send an introductory letter -include with the survey, a good cover letter -include a SASE and consider a token monetary reward -send follow-up emails -good physical appearance -include clear instructions
Rules for Survey Questiosn
-should be clear -should be answerable -should be relevant -avoid double-barrel questions -avoid biased questions and terms
Consider the order of questions:
-start with easy, interesting questions with broad appeal -generally the questions should be sorted into broad thematic categories, which are separate sections in the questionnaire -be aware that the order in which questions are asked can influence the responses that you get
Unit of Observation
-the cases about which measures/data actually obtained in a sample -the researcher may collect data from individuals(the units of o.) and then aggregate and analyze the data at the group level-making the (larger) group the unit of analysis
Institutional review board (IRB)
-the committee at the university, hospital, or research center responsible for evaluating the ethics of proposed research using specific guidelines -existence required by federal regulations -all research involving participants has to be submitted to the IRB -have the authority to require changes in research methods if they feel it is necessary to protect participants
Sampling Error
-the difference between the characteristics of a sample and the characteristics of the population from which it was selected -if the sampling error gets too large, the sample will not be representative of the population, and the findings will not be generalizable
Non-response Error
-the error that occurs due to the lack of response by some sample members -when response rates are low, the sample will not be representative of the population, and the findings will not be generalizable -it's important to figure out how non-respondents may be different than respondents
Units of Analysis
-the level of social life on which the research question is focused, such as individuals, groups, towns, or countries -in most sociological and psychological studies, the units of analysis are individuals
Secondary data analysis
-the method of using preexisting data in a different way or to answer a different research questions than originally intended -usually quantitative data-surveys or data collected by government agencies
What is the ICPSR?
-the most extensive collection of social science datasets in the US outside of the federal government -housed at the University of Michigan, that includes over 600 member institutions
In addition to clearly biased questions and terms... Be aware of "social desirability bias," also termed "agreement bias" or an "acquiescence effect"
-the question should be worded so that the response choices each seem as socially approved, or "agreeable," as the others -also consider asking multiple questions with reverse coding
What are the challenges in using secondary data analysis?
-the researcher's inability to design data collection methods that are best suited to answer his/her particular research questions -one cannot test and refine the measures or methods being used on the basis of preliminary feedback -it may not be possible for a secondary data analysts to focus on the specific research question of original interest and one may need to modify their question or give up(until new data is collected) -data quality is always a concern and usually unknown(ex. much governmental data is recorded by careless workers who are not considering future research)
Famous Unethical Cases
-the tuskegee syphilis experiments(1930's-1972) -stanly milgram's obedience studies(1960s) -philip zimbardo's prison studies(1971) -laud humphrey's tea room trade(1970)
Ethical Issues
-there are usually fewer ethical issues to worry about in survey research than in experimental research or field research -it's easy for respondents to tell that it's voluntary -the primary concern is with surveys is often that data is keeping it confidential, when it is not collected anonymously -when doing panel design surveys, special care must be taken to protect identities between "waves" of data gathering
Informed...
-they know the identity of researcher -they know they are participating in a study and are fully informed about the topic and methods of it
Why is it so easy to linking dietary habits( or individual foods) to health factors but so hard to prove causation?
-they only see the food but not the outside factors that go with it -so many variables are bound to have some of correlation -long periods of time so some health thing is going to happen so there is no randomization
Questions should be clear
-use shorter rather than longer words and sentences -don't use words that not everyone knows -breaking up complex issues into simple parts(and multiple questions) reduces confusion -don't use double negatives-and try to avoid negatives altogether
Unipolar Rating
-use words to identify categories that range from low to high(can be labeled or unlabeled) -ex. extremely, very, quite, somewhat, not at all
Association(correlation)
-variation in one variable is related to variation in another variable -"A" and "B" are related, but "A" does not cause "B" i.e. remove A and B still continues, as both are being caused by C, D, or even E
Casual Effect
-when a series of concrete events, thoughts, or actions results in the occurrence of a particular even or individual outcome -an idiographic explanation also may be termed an individualist or a historicist explanation -more often used in qualitative research, these can be useful when building theory(exploratory research)
Context Effects
-when one or more questions influence how other questions are interpreted -example: question placements which questions came first
Causal Effect
-when variation in one phenomenon-an independent variable-leads to or results in variation on average in another phenomenon, the dependent variable -in this perspective, researchers who claim a "causal effect" have concluded that the value of cases on the dependent variable differs from what their value would have been in the absence of variation in the independent variable -usually experimental or quantitative -when researchers say "cause" they usually mean a nomothetic cause
What does a researcher obtain informed consent from participants?
-written consent: a statement that describes the study and formally requests participation which they sign -verbal consent: a script that is read that participants verbally agree to(usually recorded) -information sheet: when research will otherwise be anonymous this is used to inform participants. They usually click/check a box to indicate understanding or continue with the study if they desire
What three factors do you need to establish causality?
1. (empirical) associations 2. time order 3. non-spuriousness=not caused by a third variable -need to rule out all alternative explanations
Principles for the Protection of Research Participants
1. Do no harm/protect participants from harm -what kinds of harm? metal, physical, legal/economic, social, financially -is it possible to do research with no risk of harm? 2. Voluntary participation: participants choose to participate on their own free will -they know they have the right not to participate, to stop at any time, and/or skip any question they choose to -no indirect coercion, no authority used -take extra care with vulnerable populations(ex. children, students, pregnant women) 3. Informen consent
ASA has adopted a Code of Ethics which adds to the guidelines(as have other disciplines). Sociologists are also required to adhere to these professional ethical principles when conducting research. These include....
1. commitment to achieving valid results: having a commitment to achieving valid results is the necessary starting point for ethical research practice-otherwise there is no point in any risks 2. honesty and openness. The scientific concern with validity requires that scientists be open in disclosing their methods and honest in presenting their findings 3. respect the rights, dignity, and diversity of others 4. be socially responsibly in their communities and contribute to the public good
To reduce measurement error, researchers need to
1. present clear and interesting questions in a well-organized questionnaire 2. tailor questions to the specific population surveyed -these both encourage respondents to answer questions carefully and to take seriously the request to participate in the survey
What are some reasons why it is a good data collection tool? What are some reasons why it is not?
Good: it is usually accurate, self reported, good visuals of day to day , very valid if written in for a long time Bad: people might forget to write something down or don't want to write down
Example of a nomothetic causal effect
Individuals arrested for domestic assault tend to commit fewer subsequent assaults than do similar individuals who are accused in the same circumstances but not arrested
Time Order
a criterion for establishing a causal relation between two variable; the variation in the presumed cause( the independent variable) must occur before the variation in the presumed effect( the dependent variable)
Nonspuriousness
a criterion for establishing a causal relation between two variables; when a relationship between two variables is not caused by variation in a third variable
Association
a criterion for establishing a nomothetic causal relationship between two variables: variation in one variable is related to variation in another variable
Cohort Study
a longitudinal study in which data are collected at two or more points in time from individuals in a cohort
Statistical Control
a method in which one variable is held constant so that the relationship between two(or more) other variables can be assessed without the influence of variation in the control variable
Computer Assisted Personal Interview (CAPI)
a personal interview in which the laptop computer is used to display interview questions and to process responses that the interview types in as well as to check that these responses fall within allowed ranges
(Causal) Mechanism
a process that creates a causal connection between two variables
What is a "food diary"?
a self reported food journal of what you eat each day at all times of the day during real time
Matrix Question
a series of questions that concern a common theme and that have the same response choices
Likert Item
a statement followed by response choices ranging from "strongly agree" to "strongly disagree"
Group-Administered Survey
a survey that is completed by individual respondents who are assembled in a group
Mixed-mode Survey
a survey that is conducted by more than one method allowing the strengths of one survey design to compensate for the weakness of another and maximizing the likelihood of securing data from different types of respondents; for example, nonrespondents in a mailed survey may be interviewed in person or over the phone
Computer Assisted Telephone Interview (CATI)
a telephone interview in which a questionnaire is programmed into a computer along with relevant skip patterns and only valid entries are allowed; incorporates the tasks of interviewing data entry and some data cleaning
What is the Food Frequency Questionnaire?
asking people how often they eat particular foods and what serving size they usually consume
In-person surveys(quantitative interviews)
biggest benefit? -response rates are very high -questionnaire can be longer and have more complex questions -respondents understanding of the question can be monitored -physical/social circumstances can be monitored and distractions reduced biggest concerns?
Group Administered Surveys
biggest benefit? biggest concerns? -not usually feasible to use-where is it used? -respondents may feel coerced to participate and may answer less honestly-what to do?
Omnibus Survey
covers a range of topics of interest to different social scientists, in contrast to the typical survey that is directed at a specific research question example: GSS
Technology is making this problem worse-how?
no landline phones, caller ID, to many internet surveys
Panel Conditioning
participants might change their attitudes, emotions, and behavior due to the questions being repeated asked during data collections
What are some issues involved in using it to capture valid data?
people underreport what they consume, not easy to remember everything you eat, bad questions->involve to much time/seasonal questions
Subject Fatigue
problems caused by panel members growing weary of repeated interviews or become so used to answering questions they start to answer throughtlessly
Measurement Error
the error that occurs when the data is not an accurate portrayal of what we tried to measure
Why is randomization an important part of research design?
the greater the number of cases assigned randomly to the groups, the more likely that the groups will be equivalent in all respects
Cover Letter
the letter sent with a mailed questionnaire that explains the survey's purpose and auspices and encourages the respondent to participate
Panel Attrition
the loss of subjects from a study because of disinterest, death, illness, or inability to locate them
Randomization
the random assignment of cases (elements), as if by the toss of a coin
Questionare
the survey instrument containing the questions in a self-administered survey(online or paper)
Skip Pattern
the unique combination of questions created in a survey by filter questions and contingent questions
