Research Methods - POLS 3950 - Final Exam

Limitations of Science

1. Humans are complex meaning that our behavior is complicated and may not form a 'regular' pattern 2. Trying to identify and measure such complicated behavior is a challenge a. Things mean different things to different people; defining happiness or democracy, etc. 3. The idea of determinism is problematic because it negates the idea of free will. By trying to find causes, the scientific approach assumes that a deterministic relationship exists. Yet, humans intrinsically have free will so if we can make our own decisions, then how do we deal with this idea that relationships are deterministic? 4. The scientific model was designed for the natural sciences (i.e. physics) there is greater agreement on key concepts like heat, mass, etc. whereas in politics there are many highly contested concepts (i.e. freedom, democracy, globalization). Also, research in the natural sciences is largely done in labs which is a controlled environment. Compare this to doing research in the world of politics which is not controllable. 5. Ideology, specifically feminism, postmodernism and Marxism, suggest that positivism defends the power relations structure. By objectively trying to explain power, some argue that this is done in defense of current distributive patterns of power whereas feminism illustrates gender and how political/economic/social institutions are gendered. a. The positivist model is devoid of any critical investigation of how power is distributed in our society. So if you're a Marxist, you're not going to use positivism because it doesn't explain power relations. There is a question of whether the positivist model just enforces the current social model

• Example of Internal vs. External Validity: Welfare to Work Training Program

o Jane started a work training program for social assistance recipients on a voluntary basis. Social assistance recipients can voluntarily sign up for this six-month work training program. You decide to examine whether this welfare to work program is successful. o Your data reveals that 95% of the people who took Jane's program found employment. You also decide to examine the ability of social assistance recipients who did not take Jane's program to reintegrate into the workforce. Your data reveals that 5% of these individuals found employment ♣ Internal Validity: Did the work training program (X) really lead to a change in finding employment (Y)? Or, since the people who volunteered to be part of Jane's program, would they have found employment anyway? Is it possible that the people who volunteered for the program may not have barriers to prevent them from finding employment (I.e. maybe they have no children so access to affordable and accessible child care isn't a barrier for them but others may encounter this barrier?). Is it possible that something else (another X) caused Y to change? ♣ External Validity: Can we really apply this 95% success rate to all work training programs? What if the programs were mandatory? What if the program is mandatory without any additional supports to assist reintegration into the workforce?

Exercise: What is the Hypothesized Relationship? • Political conservatism/support for balanced budgets • Support for national child care program/gender • Completion of a government-sponsored job training program/full employment • Political party membership/age • Liberalism/support for social welfare programs

o Political conservatism/support for balanced budgets ♣ No connection between increase - increase in Political Conservatism X results in increase in support for balanced budget Y o Support for national child care program/gender ♣ The direction of the relationship is unclear - increase in Gender X is unclear if it results in increase in support for national child care program o Completion of a government-sponsored job training program/full employment ♣ No connection between increase - increase in Completion of a government-sponsored job X results in increase in Full Employment Y + o Political party membership/age ♣ No connection between increase - the increase in Age X results in increase in Political party membership Y (cannot be reversed, age can't change) + o Liberalism/support for social welfare programs ♣ No connection between increase - the increase in Liberalism X results in increase in support for social welfare programs Y

Qualitative and Quantitative Approaches: The Debate

• "It makes little sense to have researchers who specialize in qualitative research yet do not appreciate there are situations when such an approach is either inappropriate or less useful than the alternatives...researchers need to be trained to recognize the limitations of their approach or sufficiently multi-skilled to be able to shift approaches when the question at hand calls for it. Most practicing researchers recognize that the dichotomy of quantitative and qualitative research is a false one, yet it remains among some teachers of research methods."

Ordinal Measures

• "Ordinal measures" is the name given to a set of numbers assigned in a particular order to the categories of a variable because the ORDER means something • Example: level of education o 1 = less than high school o 2 = high school graduate only o 3 = some post-secondary o 4 = post-secondary degree • In this instance as the number increases the level of education is higher, that is, there is an order to the categories • The distances (I.e. the intervals) between the values are not always equal. • Example: (page 216 Jackson and Verberg) What is the size of your community? o Under 99,999 o 100,000 to 999,999 o 1 million or over • Example: How important is religion to you? * o Not very important o Fairly important o Very important o Most important aspect of my life

Research Is...

• "Research is not solely about having a knowledge of methods. It is about the application of method. Essentially it is problem solving."

Exercise: Assess the Reasoning of These Statements

• "The more firefighter that appear at the site of a fire, the greater the extent of the fire damage" • "The greater the radio airtime devoted to country music, the greater the number of suicides" • "Murderers are imprisoned because they have been convicted of murder"

Exercise: Are These Hypothesis Statements Helpful? How Would You Revise Them?

• "Women in a liberal democracy like Canada enjoy more political rights and freedom than women living in a totalitarian regime" o Of course, it is true. We're looking for the things that cause change, this is almost a fact. We have to allow room for our hypotheses to be refuted. We need cause and effect. There's nothing to search for in this statement. • "Age is negatively associated with legalization of marijuana" o Negatively associated in what way? This isn't clear or directional. How does age effect it? Young? Old? • "Supreme Court Justices in Canada should not be appointed to their positions" o It's normative statement (the word should). It's not a hypothesis, it's not entirely empirical o Once you have empirical research you can make normative statements. But you can't do that with hypotheses. This isn't a hypothesis. • "There are more grade 1 students enrolled in French immersion schools than unilingual English schools in Manitoba" This is a fact, there's no cause and effect

Null and Alternative Hypothesis

• A hypothesis is a hunch of what you think is going to happen between an independent and dependent variable • Researchers state two hypotheses: o Null hypothesis: is a statement of a relationship you want to reject. This is also expressed as H0: o The Alternative (or research) Hypothesis is your actual prediction or hunch. This is also expressed as HA:

Rules for Minimizing Bias

• Alert key players (communication and education) • Avoid sexism and racism (in assumptions for example) • Clearly identify roles (researcher, assistant) • Be skeptical (particularly websites / newspapers) • Read literature cautiously • Distinguish advocacy from pure research • Be accepting of all interviewee responses as valid because all answers should be considered valid (you should never assume that there are only certain answers) • Specify data analysis procedures BEFORE to avoid data mining • Check for coding errors • Report the extent of data massaging

Validity

• Asks the question: how well am I measuring the concept according to commonly accepted definitions of the concept (I.e. liberalism, socialism). In order for your data collection instrument to be valid, it has to measure what it is supposed to be measuring. • What is a commonly accepted definition of the concept? A lit. review helps us to derive a common definition. If your concept is liberalism, you need to have a solid definition of this concept keeping in mind the differences between classical liberalism and welfare liberalism • Examples: o Jackson and Verberg's example page 351 of socio-economic status (SES) measured by income. ♣ Concept: SES which is defined as hierarchical continuum of respect and prestige ♣ Measure: What is your income? ♣ BUT, the town's religious leader is highly respected but poorly paid. Conversely, the drug lord is rich, but has little respect and prestige in society. Money is not a good tool to measure SES

Problems with sources of non-scientific knowledge to develop conclusions:

• Authorities: may talk outside their field of knowledge and outside their credentials. Where did they receive their credentials? May be biased, may falsify data. How did they acquire their data? o Example: "Dr." Grey who wrote the book Men are from Mars and Women are from Venus. He has an unaccredited PhD degree from correspondence in 1982 from a defunct university • Media: do not fully investigate all aspects of an issue (i.e. the famous "ten-second sound bite" and distort the issue). Only provide short stories and with the concentration of newspaper ownership in Canada fewer opinions are being heard. Journalists may be biased and may be under pressure to 'sell', and not 'educate' • Intuition: a form of limited information and is not always right. You don't understand why you have this feeling but you have a gut-reaction about a certain issue • Stereotypes: these are generalizations about a group of people based on limited direct knowledge. These stereotypes do not hold true and are harmful. o Example of a stereotype: French Canadians are snobs • Faith: these are beliefs that cannot be proven one way or another (i.e. concepts of Heaven and Hell). These beliefs are acquired at an early age and are not generally challenged but accepted.

Who Decides What is Valid?

• Babbie (2007: 148) states that "social researchers should look to their colleagues and to their subjects as sources of agreement on the most useful meanings and measurements of the concept they study. Sometimes one source will be more useful, sometimes the other but neither should be dismissed".

Bias

• Bias in social research can most easily be described as 'distortion'. Bias is distortion and is always systematic but not always deliberate. • It can occur AT EVERY STAGE of research and in many different forms • It is impossible to be completely objective; realistic goal is to be aware of potential bias and attempt to minimize.

Exercise: Assess the Validity of These Measures

• CONCEPT: support for feminism in Canada o Indicator: membership in a feminist organization o Question: Do you hold membership in the National Action Committee on the Status of Women? • CONCEPT: experiences with victimization o Indicator: reported crimes to police o Question: what is the crime rate in community 'X' as drawn from filed police reports? • CONCEPT: Religiosity o Indicator: attendance at a religious institution o Question: How often do you attend service at your religious institution each week? • CONCEPT: Compassion o Indicators: ♣ public displays of compassion; ♣ donating; ♣ Supportive of changes to social policy to help the underprivileged. o Questions: ♣ Do you cry during dramatic movies? ♣ Do you give money to people begging on the street? ♣ Do you believe that people on welfare should receive significant increases in benefits?

Causality...

• Causality: where an independent variable, X, is a cause for a change in a dependent variable, "Y". o X → Y • In a cause-effect relationship, the independent variable is the 'cause' and expressed as an "X" • The dependent variable (Y) is the effect o Example: weight as the dependent variable (Y) and calories and exercise as independent variables (X1 for calories and X2 for exercise)

Construct Validity:

• Construct Validity: The degree to which a measure actually relates to a variable as set out in a theoretically derived hypothesis. Jackson and Verberg (355) state that the way to think about construct validity "is to recognize that it is based on inductive evidence. If one finds evidence to support a theoretically derived hypothesis, then this would indicate that one's measures have construct validity." o Example*: Marital satisfaction. You have developed some theoretical expectations about how marital satisfaction relates to other variables (I.e. satisfied spouses are less likely to cheat). If the measure of 'cheating' relates to your theoretical expectations then you have construct validity. But, if satisfied partners (married) are just as likely to cheat as are dissatisfied partners, then the construct validity is challenged.

Content Validity:

• Content Validity: how well measures cover the range of meanings contained in a concept Example*: How law schools look beyond just the LSATs when admitting students. Canadian law schools look at LSATs, GPA, performance on the interview and some may look at reference letters. Students with the best overall measurements are admitted

Criterion Validity:

• Criterion Validity: Established over period of time to see if indicators are successful in reflecting reality. o Example: LSATs measures the ability of a student to successfully complete law school. To see if LSATs have criterion validity, we need to compare LSAT scores with the LSAT writer's success at completing law school. There would be a high degree of criterion validity if the students who scored well on the LSATs also had high GPAs in law school. But, if students who scored well in the LSATs, but had lower GPAs in law school, then there would be a lower degree of criterion validity. *

Exercise:

• Decide which technique to use: qualitative or quantitative or would you use both and how. Do you have any concerns about your choice? 1. The degree of feminist values held by current Members of Parliament in Canada 2. Why some people do not vote in Canadian elections 3. Whether a new policy to provide tax credits to caregivers of family members with a disability has been successful

Descriptive Versus Explanatory Research

• Descriptive Research: attempts to accurately describe the population in question. Concentrates on who, what, where, when, not why o Example: who is more likely to support increases to social welfare - men or women? • Explanatory Research: seeks to understand and explain relationships. Concentrate on 'why' not who, when, where, or what. o Example: why are women more likely to support increases to social welfare?

Respectability of the Research: Quantitative vs. Qualitative

• Despite these differences, qualitative and quantitative approaches attempt to generate knowledge in a legitimate and trustworthy manner. • To do so, both rely on a set of criteria. Although these criteria have different labels, they attempt to illustrate the trustworthiness of the research. Quantitative Validity Reliability Objectivity (i.e. impartiality) Qualitative Credibility Dependability Confirmability

Quantitative and Qualitative Approaches to Empirical Research

• Empirical research involves observations and there are two key ways to achieve this: qualitative and quantitative approaches. • Quantitative approach involves a larger quantity of cases (i.e. may involve politicians, feminists, judges, etc.). Because of time and financial constraints, there is no in-depth study. Instead a data collection instrument is developed and applied 'as is' to a large number of cases. Breadth over depth. • Qualitative research emphasizes quality or richness of the data with an emphasis on learning more about a smaller number of observations. This approach emphasizes depth, not breadth and is less structured than quantitative research in order to allow the research to diverge into a new area of analysis (i.e. probe when necessary). As a result, this approach does not yield generalized statements about the population being studied.

The Response Rate: What You Can Change

• Ensure it is professional and visually appealing (I.e. spelling, grammar, lots of white space, no cheap photocopies). Include a SASE. • Include a cover letter to explain the research, identify sponsoring agency for ethical reasons but also to add legitimacy and worthiness to the project. Explain benefits of participation, how respondents were selected, instructions on how to complete questionnaire and a statement about confidentiality/anonymity. Convince the person there is value in participating (I.e. Our research depends on people like you!) • Provide monetary incentives to increase response rate Ensure follow-up (I.e. postcard reminders, registered mail, or phone call). Send a schedule of reminders: two weeks, four weeks, six weeks. Consider re-sending the survey if needed.

Examples of an Index

• Example: Index of political activism: o Ask people which of the following actions the respondent has taken part in: ♣ wrote a letter to a politician; ♣ signed a political petition; ♣ participated in a protest; ♣ gave money to a political cause; ♣ gave money to a political party or candidate; ♣ wrote a letter to the editor ♣ Give one point for each one checked and you now have an index that ranges from 1 to 6. The more point an individual accumulates the more politically active they are • Example: Index of conservativism from CPEP o The conservative index ranges from -12 (ultra liberal) to +12 (ultra conservative). Two points applied to 'strongly agree'; one point to 'somewhat agree'; minus one point to 'somewhat disagree'' and, minus two points to 'strongly disagree'. Respondents were asked the following six questions: ♣ Government should leave it entirely to the private sector to create jobs; ♣ Government regulation stifles personal drive; ♣ People who do not get ahead should blame themselves, not the system; ♣ Society has reached the point where women and men have equal opportunities for achievement; ♣ This country would have fewer problems if there were more emphasis on traditional values; and, ♣ If Aboriginals would only try harder, they could be just as well off as everyone else.

External Validity:

• External Validity: Given the completion of your research, can your results really be applied to other groups in society? o Example: Welfare to work training program

Different Types of Validity

• Face Validity • Criterion Validity • Construct Validity • Content Validity • Internal Validity • External Validity

Face Validity:

• Face Validity: Where your measurement, on the surface, reflects the concept. Does the indicator make sense to you? If yes, you have face validity. o Example: If you are investigating beliefs in conservatism and you ask the question: "Do you support taxes being used to pay for NHL hockey teams?" This question does not reveal validity because it doesn't measure it. A conservative may or may not support this and this doesn't reveal anything about conservatism

Exercise: Identifying Indicators

• In your group, identify indicators for three of the following variables. Provide the wording for the survey question resulting from your indicator including the response categories. o Support for environmentalism o Political interest o Democracy o Human rights o Support for feminism o Liberalism o Poverty

Indexes

• Indexes are composite measures that add together the responses to several questions to come to some overall measure of a concept • Important for content validity to employ several questions that touch upon the many different aspects of the concept in question (like nationalism, feminism, alienation)

Other Variables

• Intervening Variable: links the independent to the dependent variable: • Example: In the US, income (X) and a long life (Y) may have an intervening variable of access to health care (I). Having money, by itself does not mean you live longer, but it buys medical care that can help you do so IMAGE 3 • Antecedent Variable: where a variable causes variation in the independent variable, which will then influence the dependent variable • Example: parent wealth (A) may cause a change in the student's university test score (X) and thus their entrance into university (Y) IMAGE 4

Interview Schedules

• Interview Schedule allows much more flexibility and autonomy for the interviewer as s/he is not bound to read out questions word for word. Rather, s/he is given a list of areas to probe (this requires a talented interviewer!) • The interview schedule may be expensive (interviewer time, prep, training and possible travel time). • Reasonably high response rate. • Interviewer effect • Better rapport than phone surveys • Some respondents may not be accessible.

A Helpful Hypothesis Statement is...

• Is clear. Both the IV and DV are clearly articulated • Provides directionality. The connection between IV and DV is outlined • Is falsifiable. There is evidence, or at least you can imagine there is evidence, that exists that would not support your hunch Examples... • If we suspect that age (as an independent variable) is influential in rates of volunteerism, then our hypothesis become: o Null Hypothesis (or H0:): There is no relationship between age and volunteering o Alternative Hypothesis (or HA:): As age increases, the rate of volunteerism also increases (notice the indication of the relationship between X and Y which is important in an alternative hypothesis statement)

The Qualitative/Quantitative Approach

• Is one approach better than the other? Any research topic can be researched with either approach although some topics (i.e. voting) may lend themselves to one approach over another. Subject matter should not, however, drive the selected approach. • Few researchers are 'purely' qualitative or 'quantitative and prefer to opt for both as different tools in the toolbox. The positivist / post-positivist divide appears to be lessoning within the literature and growing expectations that researchers learn different methods depending on the specific research question.

Operationalization: Things to Consider

• Need to think about how much variation and precision you want in your responses. o Example: Do I need to know exact income level or will general categories suffice? • The more precision you need the greater the level of error. If you ask what was your 2009 tax refund down to the penny, many people will not remember and round off leading to a measurement error o Example: Do I need to know exact age or will categories be okay? • Want to make sure you include all possible categories and/or full range of variation if necessary o Example: can't allow only two categories such as strongly agree or strongly disagree. Why? Some people may be neutral and you will force them to give you an invalid answer • Need to think about the dimensions of the variables you want to measure o Example: if you are interested in corruption in government, you need to decide if you want to measure how much there is, whether people think it exists, how big a problem they think it is, what causes it, and what should be done about it. • Also need to ensure that for every variable: o Categories are exhaustive o Categories are mutually exclusive • Categories are exhaustive: You must include all possible answers o Example: What is your age? ♣ Under 20 ♣ 21-40 ♣ 42-60 ♣ 61 and over ♣ *WRONG: 41 is missing, so it's not exhaustive • Categories are mutually exclusive: no duplication o Example: What is your income? ♣ Under $20,000 ♣ $20,000 to $40,000 ♣ $40,000 to 60,000 ♣ $60,000 and over ♣ *WRONG: if your income is $40,000, which box do you choose? Answer categories are not mutually exclusive

Nominal Measures

• Nominal measures are those that offer names or labels to categories that are exhaustive and mutually exclusive but for which little else is true. Numbers might be assigned to these categories (to distinguish the categories) but the numbers don't mean anything in and of themselves. • Nominal measures example: (Page 216 in Jackson and Verberg) o Question: What is your place of residence? ♣ Atlantic Canada ♣ Quebec ♣ Ontario ♣ Western Canada ♣ Other • Example: asking about religion using nominal would include answers such as: o Roman Catholic o Protestant o Sikh o Hindu • What is wrong with these answer categories? They're not exhaustive, what if someone is something other than what is listed

Advantages of Open-ended and Close-ended Surveys

• Open-ended o Advantages ♣ Allows for valid response ♣ Allows for more possible responses the researcher might not have considered • Close-ended o Advantages ♣ Uniformity of response ♣ Easy to code ♣ Less intimidating ♣ Not too time consuming for respondent

Disadvantages of Open-ended and Close-ended Surveys

• Open-ended o Disadvantages: ♣ Time consuming to code ♣ Difficult to interpret (I.e. hand writing) ♣ Difficult to compare across ♣ May intimidate some respondents • Close-ended o Disadvantages: ♣ Researchers may not include all potential responses. ♣ Use of 'other' category not writing) successful ♣ Possible responses may not respondents fully capture respondent's beliefs.

Order of Questions for Survey Questionnaires

• Place easy to answer items at the beginning and increase difficulty throughout • Place non-threatening items at the beginning (don't scare them). Demographic information (I.e. age, income) are always at the end of a survey • Open-ended questions should appear at the end since they involve more work • Group questions into sections to provide a flow to the survey and easier interpretation • Vary question format as you want to avoid 'response set' which is a number of questions that are very similar and the respondent may get bored/lazy bcz they all have the same format

Pure Versus Applied Research

• Pure Research: involves a researcher who seeks to understand social behavior but not interested in advocating societal change. There may not be any practical value gained from pure research • Applied Research: involves a researcher who engages in research to make societal changes and thus takes on an advocacy role

Sources of Bias

• Researcher affect refers to the scenario where a researcher falls in love with a particular explanation and uses procedures to support the preferred explanation. Think of the Fraser Institute where a researcher is paid to come up with 'right-wing' work if s/he wants further work - this is a self-interested motive. • Social desirability bias distorts the validity of the data because respondents may over-report what is socially desirable and vice versa. Examples: voting, volunteering, donating to registered charities

Define Sample, Population and Sampling Frame

• Sample: a segment of the population that you study to gain information about the whole. • Population: the collection of individuals, communities or nations about which one wishes to make a general statement (I.e. students who reside in the U of M residences between Sept 2006 and April 2007.) • Sampling frame: list of individuals, nations, etc from which you select your sample. The telephone book is not a good sampling frame as it omits people who have moved or with unlisted numbers (I.e. list of students, list of employees, list of voters)

Sampling

• Sampling is the process of selecting observations for your research project. • You sample to save time and money - why sample 10 million people when sampling 2,000 will suffice if the sample is properly constructed

Appearance and Length for Survey Questionnaires

• Self-administered: o Don't want it to be too long but don't cram as many questions on one page either. o The design should be pleasing to the eye with a nice font and typeface o Use good quality paper - no photocopies! • Telephone o Difficult to judge length - Jackson says 20 minutes o Interview / Internet - no concrete rule

Why Use Indexes?

• Some issues are quite complex (I.e. views on abortion - do people have the same views when the mother's life is threatened or the child was conceived by rape or if the child was economically inconvenient?) Thus, a general question lacks precision because someone may be opposed to abortion in general but in favor if the woman was raped or her life was in danger. • Multiple items allow you to evaluate reliability and validity than just using a single indicator.

Errors in Reasoning ****

• Source of spuriousness model: where a variable (S/S) has an effect on both X and Y such that there is no relationship between X and Y and thus illegitimate or phoney o Example: provided by Jackson and Verberg: When exploring the relationship between socioeconomic background and choice of nontraditional program by female students, consider the possibility that rural/urban background is the source of spuriousness. Does coming from an urban vs. rural background influence parents' socioeconomic status as well as university program preferences? CAN'T IT JUST BE REVERSED SO THE S/S IS X? IMAGE 5 • Ecological fallacy or faulty thinking when we draw conclusions or interpret data erroneously o Example: faulty thinking in terms of scale. If your research reveals that the average income for St. Vital and St. Boniface is $60,000 and you conclude that most residents in those communities have incomes of $60,000 this is faulty thinking WHY IS THIS FAULTY? • Tautology: circular thinking or reasoning that predicts nothing o Example: consider the comment by former PM Jean Chretien in relation to the sponsorship program: "The proof is the proof. And when you have good proof, it is proven" WHAT DOES THIS MEAN THOUGH?

Interviews

• Structured Interview is an interview, face-to-face, where the interviewer merely reads the questions as worded. This is like a conversation but not really since the interviewer doesn't talk outside of the questions being asked. S/He controls the topic, asks the questions, and does not share opinions with respondent. Goal is to ensure that different responses are due to differences across respondents rather than to changes across interviews and/or the survey instrument. • These interviews have a reasonably high response rate, very few skipped questions, clarification is possible. • Better rapport than in telephone interviews More expensive, especially if you need to hire and train interviewers. May be travel costs • Interviewer effect can set in (I.e. body language read by respondent) • Some people are hard to get to (I.e. people in secure apartment blocks, gated communities, rough neighborhoods)

The Response Rate: What You Cannot Change

• Studies show that students, employees and military personnel are more likely to complete questionnaires than rest of society • The type of agency conducting the research influences whether people complete the questionnaire (I.e. Stats Canada) • Salience of the topic is another factor. If people think your topic is interesting and important then your response rate will likely be higher.

Surveys

• Surveys are a way of collecting information by having participants complete a questionnaire. Surveys are best at collecting information from a large sample size but may not be best for all research designs. • Surveys are generally associated with positivist approach to knowledge. • The questionnaire is comprised of a set of questions. You must craft your questionnaire with care (I.e. don't write the questions with language only a graduate student can understand).

Surveys: Population vs. Sample

• Surveys are employed with a 'sample' drawn from the 'population'. o Population: The collection of individuals, communities or nation-states about which you want to research and make generalizations. o Sample: A segment of individuals, communities or nation-states selected from the population that you will research in order to represent the population. o Surveys are thus administered to the sample, not the entire population. Why? The financial costs associated with surveying everyone in the population is prohibitive so we rely on the sample to represent the population. o Sample must reflect the population (i.e. if the population is 52% women, then the sample must represent this)

Science as a Source of Knowledge - 5 Characteristics of the Scientific Model

• The five characteristics of a scientific approach to knowledge acquisition are: 1. Science is empirical meaning that science is based on direct observations of the world and not on speculation. This means that some topics that cannot be observed do not fall within the scope of scientific inquiry (for example, whether a higher being(s) exists because direct observations are not possible) ♣ The distinction between 'empirical' and 'normative' modes of inquiry • 'Empirical' is concerned with 'what is' and 'why', but is not concerned with 'what should be', which is normative • For example, the scientific mode of inquiry can tell us how much and why the Manitoba gov. spends on social welfare, but it cannot tell us if this spending is good or bad because this a value question, which exists in the realm of normative modes of inquiry • That said, research related to the politics is concerned with the values of 'what should be', meaning that values are present in the research. Researchers generally examine relationships empirically in order to make conclusions about 'what should be' 2. Science is systematic such that the 'procedures used by scientists are organized, methodical, public, and recognized by other scientists' so it is not random or haphazard. Scientists disclose all the procedures used in order to arrive at their conclusions. Also, there needs to be replication of the research (i.e. repeating the research many times to determine if the same findings will appear) 3. Science searches for causes of events. This is based on the assumption that the world is ordered and patterned. In an orderly and patterned world, there are reasons for why things occur and scientists attempt to understand these causes ♣ Underpinning this search for causes is the belief that there is regular patterns of behavior. For example, the behavior of war has occurred repeatedly over time so we can search for the causes of war recognizing that not all wars are identical ♣ With a search for causes comes the idea of probability. If we can identify those causes that determine a specific type of behavior, we can predict behavior ♣ While we can predict behavior, it is problematic in research related to politics because human behavior is not always predictable or rational. In addition, it is difficult to identify and measure all the causes of certain forms of behavior ♣ Human behavior is not always as patterned or predictable or rational ♣ Predictive power in social sciences won't hold up as well as natural sciences ♣ Always looking for causes of why something is happening, and you can then predict what will happen next time 4. Science is provisional meaning that findings are not seen as ultimate or irrevocable. Instead findings are open to being questioned and possibly refuted 5. Science is objective meaning that researchers avoid their personal values and biases when conducting research. But, is objectivity really possible?

General Introduction to Survey Questionnaire

• The introduction should describe the study's purpose, the group/institution undertaking the research, outline confidentiality of the responses, and appropriate time to complete survey • You should try to explain why the survey is important and why the respondent should take time out of their day to finish the questionnaire • Example: From the 2008 Canadian National Election Survey: o "This is the final component of the National Election Study. We appreciate the assistance you have provided us in the study and hope you will complete the questionnaire as soon as possible. All responses to this questionnaire will be kept confidential. Please return the questionnaire in the enclosed self-addressed envelope. No postage is required"

Positivism (Or the Scientific Paradigm)

• The paradigm of choice in the physical sciences. Adopt 'science' in the study of social issues and behavior. Inquiry based on logic, statistics and use of established methodological techniques • Represents a move away from religion and metaphysics towards logic and rationality expressed through positivism. Associated with the work of Auguste Comte

Things to Remember About Survey Questionnaires

• The response categories should be exhaustive; you might want to include the set of responses a category labeled: "Other: _______" or "Please specify____" which allows them to fill in the appropriate response • The response categories should also be mutually exclusive (I.e. only one response) unless you specify that multiple answers are okay. • Search for previous questionnaires that asked the same thing (or similar) but you must be critical of these surveys. • Ask yourself: why am I asking this question and is it valid and reliable? ¢ • KISS (keep it simple, silly). Use simple wording to avoid confusion and ambiguity. No jargon and no acronyms unless universally accepted (remember, the Mayor of Toronto stated he never heard of the WHO!) Translation is a concern. • Avoid using negative items as this leads to confusion (example: do you disagree that universal health care should not be available?) • Avoid biased terms/items (example: do you agree that people on welfare deserve more support). • Respondent should not feel that there is a wrong response • Prestige bias is the use of a well-known person (the Prime Minister) in the question that may lead respondents to greater/lesser support than they would otherwise. • Avoid leading questions such as wording that a particular response (example: should people be allowed to kill others whom they think will be better off dead?). • Avoid double-barreled questions that asks more than one questions (example: Do you support a universal health care system and higher taxes?) • Avoid extreme questions that will not result in variation of response (Example: In a country like Canada, the assassination of a person holding political office is unacceptable in order to bring about political change). This brings out a homogeneous response - we are looking for variation • Pre-test your questionnaire to avoid problems with language and comprehension • Have a dialogue with a group of 10 people to gauge their responses to the questions. You are not concerned with their opinions about the topic, but rather their opinions with the structure of the questionnaire. • Check for response set and sufficient variation in responses • Ask if they have any questions and what they think the question is asking about • See handout titled "What's wrong with each of these questions"

Science as a Source of Knowledge

• The scientific mode of inquiry to gaining knowledge is not perfect, but it is generally viewed as superior to the other source of knowledge • Science is a method of obtaining objective knowledge about the world through systematic observation • Science tells us the rules to apply but does not tell us what we know • Often referred to positivism as a paradigm. A paradigm is a way of seeing or in more technical terms a model or frame of reference we use to organize observations. Some paradigms are difficult to acknowledge because they are ingrained in our daily life. Think about the human rights paradigm • Social scientists develop several paradigms to understand social behavior. In science, the move away from one paradigm to another paradigm is generally done out of a belief that the older paradigm is now 'false", while the newer one is 'true'. In social sciences, however, there is no true or false but more or less useful

Deductive ↓ Inductive ↑

• Theoretical Level (abstract) o Abstract explanations of behavior o Theory helps us make sense of chaos by looking at specific explanations o X has an impact on Y • Conceptual Level o You begin to define concepts as outlined in the theoretical level (i.e. here you define what 'X' and 'Y' mean) ♣ Examples: peer pressure, education, poverty, alcoholism o You build your hypothesis statements, which are hunches/predictions of what you think will occur between X and Y • Operational Level (concrete) o Where you build measures or indicators to accurately reflect your definitions in the conceptual level (i.e. how much money is spent on food may be considered a measure of poverty)

Theory and Concepts

• Theories are "sets of interrelated, abstract propositions or statements that offer an explanation of some phenomenon". This means there is a relationship between certain elements, but they are quite abstract at this stage. Theories explain why things happen. o Example: shoveling fog • Concepts are also important and are "mental constructs or images developed to symbolize ideas, persons, things or events". At the conceptualization stage, we are starting to move away from the abstract theoretical level, the more concrete level, we provide definitions for the concepts outlined in our theory to give them precision.

Limitations of Phone Survey

• There have been increasing levels of refusal to participate. Phone surveys have a lower response rate that mailed questionnaires • The questions must be simple as respondents don't have luxury of reviewing the questions more than once • Doesn't allow for in-depth probing and the interviewer may influence the response of the individual. • Not everyone has a phone (I.e. some have none; others are mobile so their phone becomes disconnected)

Contingency Questions

• These are used to identify and filter out certain respondents who are not appropriate for certain questions • Example: if you have a section detailing the voting behaviour in the last federal election, then you should only speak with people who voted. Thus, your contingency question would be: •1. Did you vote in the last federal election? o ( ) yes (go to question 2) o ( ) no (skip to question 3) o ( ) not eligible (skip to question 3) o ( ) don't recall (skip to question 3) •2. Which political party did you vote for? • ( ) NDP • ( ) PC etc.

Exercise:

• Think about what kind of independent variables you would use to explain the dependent variable and build null and alternative hypotheses and a causal model: o Factors affecting professional versus traditional career orientation among women ♣ Parental Education X1 • NULL: There is no relationship between levels of parental education and the amount of professional women • ALT: An increase in parental education results in an increase in professional women in the workplace ♣ Rural Community X2 • NULL: There is no relationship between being raised in a rural community and the amount of professional women • ALT: An increase in the number of women living in a rural environment results in an increase in professional women in the workplace ♣ Family Income X3 • NULL: There is no relationship between family income and the amount of professional women • ALT: An increase in familial incomes results in an increase in professional women in the workplace

Reliability

• This is the other quality of measurement method that suggests the data collected on the first round will be repeated by the same respondent in the second round, third round, etc. o Example: weight scale - friend or foe? o Example: measuring volunteering - can people reliably recall all volunteer activity? • How can you be certain that the measures are reliable? 1. Re-testing the survey 2. Asking one question early on in the survey and then re-word the question and ask again later on 3. Verify through some external source (if available)

Simple Random Sample

• This is where every person (or unit of analysis like a nation) in the population has an equal chance of being selected for the sample. o I.e. similar to putting names in a hat and drawing • This simple random sample is more likely to be done with the use of a random numbers table • Steps: 1. Number the individuals in the population 2. Decide on sample size required 3. Go to random numbers table and randomly pick a starting point 4. Select cases as they appear in the numbers table • Number of columns in the numbers table to use depends on the number in your sampling frame (should equal the number of digits in the sampling frame count) • If a number comes up twice, ignore the second one • If a number is larger than the number in your sampling frame - ignore it • Continue until the sample is done.

Phone Surveys

• This method dominates the market research but internet surveys may soon dominate (?) • Telephone surveys can be cheaper and more efficient that conducting face-to-face interviews. The development of computer assisted telephone interviewing (CATI) is especially helpful for the researcher. The computer selects a telephone number at random and dials it (where random-digit dialing avoids the problem of unlisted numbers) • Cheap (no travel) • Quick if you have many telephone interviewers • May be best for a hard to reach target group (I.e. doctors)

Mailed Survey

• This method is self-administered so the respondent answers it themselves • Pros: cheap and makes contacts with a large number of people. May have a low response rate but researchers have some room to increase that. • Response rate: the percent of delivered questionnaires (minus the incorrect mailing addresses) that are returned (completed). a. RR= # responded divided by # sent out b. Example: 50/200 is a 25% response rate

Individual or Group Administered

• This method is self-administered so the respondent answers it themselves (i.e. nurses working at a hospital or administered to an entire class) • Questionnaire is personally delivered to individual or administered to a group in one place at the same time - more expensive than a mailed survey. • Provides opportunity for clarification and answering of questions • Response rate a little higher • Arrange for pick-up of individual surveys helps ensure a higher response rate

Internet Surveys

• This method is self-administered so the respondent answers it themselves. • Becoming as popular as telephone surveys (?) • Two different forms: 1. Where the questions are emailed out 2. Where the questions are placed on a website • Methodology still under development but refined significantly. Two concerns exist: o Sample representativeness o Multiple response

Present-Absence Questions (Surveys)

• This type of question requires the respondent to 'check off' the items which apply to them only. • Example: On a guilty verdict, for which of the following situations would you support the use of capital punishment? (Check off as many as appropriate) o Manslaughter___ o Premeditated murder___ o Rape___ o Murder of a police officer___ o Murder of a prison guard___ • What is wrong with this categorization? o The last two could fit within the first two categories, and it prioritizes the last two categories above other murders

Sampling Disasters...President Thomas Dewey

• Thomas Dewey was hailed by the infamous George Gallup as the next American President defeating Harry Truman in 1948. History tells us that Truman was elected President so what went wrong? There are several factors: o Pollsters stopped polling in the beginning of Oct. even though Truman enjoyed more support as the campaign went on; o Voters were largely undecided throughout the campaign and most ended up supporting Truman; o Perhaps the most important reason for this failure rests with... ♣ Gallup's polling relied on a sample that was not representative of the population. In order to sample, we need to understand the characteristics of our population. For large national polls, census data is usually relied upon and this is what Gallop used. ♣ However, by 1948, WWII sparked a movement away from rural America to urban America and thus changing the nature of the American populace (urbanites tend to vote for the Democratic Party). ♣ With the War, the last census was conducted in 1940. The 1940 census data showed more of a rural base than what actually existed in 1948. Thus, Gallop had too many rural Americans represented in his poll and not enough urban Americans. As such, he underestimated the support for the Democratic candidate, Harry Truman

How do you find the relationship between x and y?

• To come to the relationship between X and Y, you must: o Take one X and hold the remaining X's constant o You then think to yourself, if there is an increase in X, how would it impact Y o Does Y increase as well? If so, then there is a positive relationship between X and Y o Does Y decrease? If so, then there is a negative relationship • Example: if we hold exercise constant and we increase the number of calories per day, then weight will increase so there is a positive relationship between calories and weight

Types of Probability Samples

• Types of probability samples: 1. Simple random sample 2. Systematic random sample 3. Stratified random sample 4. Multi-stage sample

Instructions to Survey Questionnaires

• Use them to help respondents understand the questions asked and to complete the questionnaire • Directions can apply to an individual item, a section of questions or the entire questionnaire • Can include definitions of concepts/terms for clarification ¢Keep wording simple, simple, simple... • Example: This section contains a series of paired statements. Please read each statement and circle the statement which you believe is closer to your own opinion o Now, let's talk about your feelings about the tactics used by interest groups. An interest group is a group of people that work together to get their interests heard in government. I will read out the types of tactics used by interest groups and on a scale of 1-10, with 1 being 'completely unacceptable' and 10 being 'completely acceptable' rate the level of acceptability of each tactic.

Concerns with Surveys

• Validity: where participants may answer the questions to be seen in a positive manner but their day-to-day actions may be the opposite thus violating validity. So, what people say and how they act may be two different things. • Surveys gather data at a specific point-in-time only and are poor instruments for measuring changes over time. • Recalling past data can be difficult for participants (I.e. when did you start your first part-time job while still in high school?) There may be no other suitable way of measuring some variables though • Can only be used with a literate population • Since the questionnaire is highly structured, this prevents you from probing more in-depth on certain issues. If you want in-depth, you need to conduct a personal interview or focus groups (i.e. more open qualitative approach)

Theorization and Conceptualization Stages...

• What we have done so far is focus on building theory and concepts and give these variables definite meaning for our research project. This is the theorization and conceptualization stages

The Critical Concepts of Measurement

• When we begin to operationalize our concepts, we should be concerned with two things in particular: VALIDITY and RELIABILITY. These are critical concepts at the level of measurement.

Our Weight Example Can Be Graphed in a Causal Model...

• Where X → Y where X = independent and Y = dependent variable • Causal models: provide visual description of the model we will examine. Should include dependent and independent variables and nature of relationship IMAGE 1

Quick Review

• Where Y = dependent variable you are researching • Where X = independent variable which helps explain movement in Y • Hypothesis construction: o Negative relationships = an increase in X leads to a decrease in Y so the change occurs in different directions ♣ Increase in exercise: It will result in a decrease in Y weight. So, you put a ↓ over the arrow relationship, and a ^ in the exercise box, and down arrow in Y box o Positive relationships = an increase in X leads to an increase in Y so the change occurs in the same direction ♣ Increase in calories by 1: Person increases their caloric intake each day, then it will result in an increase in Y. Then you put a plus over the arrow, and an ↑ in each box • What if you are unsure? o If you're unsure of the nature of the relationship, put a "?" over the relationship arrow because there's divergence in the literature o Conditional Variables: ♣ I.e. Gender • Support for national child care program/gender • Women more likely because of the gendered caregiving roles they've been forced to grow up with • Conditional - going to investigate both genders (X1 women, X2 men) ♣ I.e. employment after training • Completion of a government-sponsored job training program/full employment • Finishing job program might be a "?" also because finishing the program doesn't necessarily mean that you'll get a job

Ratio Measures

• Where data can be mathematically manipulated because the "zero point is aligned with the true zero and the distances between the intervals are equal" (p. 216, Jackson and Verberg). For example, income. Jackson and Verberg state that because there is a true zero and the intervals are equal, it is accurate to say that $50,000 is double that of $25,000. o Examples: weight, age, GPAs,

Systematic Random Sample

• Where every Kth element gets into your sample where K = sampling or skip interval (population/sample needed) • Example: your population (sampling frame) is 1000 and you want a sample of 100 • K = 1000/100 = 10 (if not a whole integer then round off but this poses problems too) • Pick every 10th person on the list.

Likert-Type Questions (Surveys)

• Where the respondent is asked to indicate the strength of their agreement or disagreement with the statement made above. • Example: Canada's protection of endangered species is strong. o (1) Strongly disagree o (2) Disagree o (3) Neutral o (4) Agree o (5) Strongly agree • Or, can use a scale of 1-10 too

Rank Ordering Questions (Surveys)

• Where you instruct the respondent to indicate an ordering of their responses • Example: Please rank the three most important policy issues facing Canada. Place a 1 beside the issue that is most important; Place a 2 beside the next most important issue; and a 3 beside the next important one. o ___health o ___social policy o ___industry o ___regulatory issues o ___taxation o ___environmental policy o ___macroeconomic policy o ___trade policy

Exercise: UN Member States

• Working in pairs, answer the two questions on the handout related to the UN member states along with the handout (simple random number table).

Exercise: Sampling Size

• Working in pairs, identify what lists (if any) that could be used as a sampling frame for the following populations. What concerns (i.e. inaccuracies or omissions) do you have with your identified list? What sampling technique would you employ? Why? o Federal public servants o Hospital admissions o Manitoba voters o Canadian citizens o Vancouver heroin addicts o Quebec sovereigntists o Federal Members of Parliament

Example of Building Theory Vis-à-Vis Deduction...

• You are hired as a policy analyst for the province of Manitoba. On your first day, you are asked to examine 'who supports increases in welfare payments in Manitoba' • Your dependent variable is: 'who supports increases in welfare payments' and you must now do a thorough literature reviews and think about building a model to include the independent variables and the direction of the relationship IMAGE 2 • X1 Age • X2 Religiosity • Y Support for welfare increases ^ • What next: put arrows between them, hold x1 constant (ignore x2 for now). Think about if there a 1 unit increase in age, does it have increase in support for social welfare changes • Can you classify something as a positive variable within certain conditions (i.e. age, but only between 20-50?) o You have to treat it like a conditional variable then • X1 to Y is positive, x2 to Y is positive • Then form research hypothesis

Incentives for Focus Group Participation

• You may need to build in some 'benefits' to get people to commit to coming to the focus group (I.e. if the population is single Moms, will you provide transportation money and a babysitter?). If you are doing research with animal lovers then maybe you want to consider giving a donation to an animal shelter in their name as their honorarium (especially with civil servants who cannot accept gifts). • You must think of location (I.e. it must be easy to access and not in a remote area)

Non-Probability Sampling

• You use this type of sampling when you either don't want to or can't generalize to the entire population. • Why use this if you can't generalize? o Can't always get a list of populations o Might have limited resources (time/money) o Perhaps you want deeper answers so make a conscious decision not to generalize

Exercise: Strengths and Weaknesses of Focus Groups...

• work in groups of 2-3 to generate points for both strengths and weaknesses of focus groups. • See page 178-181 for help

How do we know what we know? Non-scientific vs. scientific knowledge:

•Non-scientific knowledge constitutes a large basis for much of what we know where things do not have to be proven to us in order to believe. •These sources include: o Authorities, media, intuition, stereotypes, faith o But what are the problems associated with relying on these sources?

1. Science is _____ A. a set of rules that help us understand the world around us B. a set of facts C. the study of empirical analysis D. the study of normative analysis

(A)

1. The goal of empirical research is to explain ____ with respect to the ____ event The goal is to understand why our research is varying ^^ toughest A. Variation, dependent B. Facts, dependent C. Variation, independent D. Facts, independent

(A)

1. Empirical political science is generally concerned with ______ (not A = normative) A. Prescribing the way things ought to be B. Discovering why thing are as they are C. The way politicians behave D. The way politicians think

(B)

1. Internal validity is the extent to which ___ A. The results of quantitative studies can be generalized B. The researcher has produced results reflective of reality C. The results of qualitative studies can be generalized D. Our variables measure our concepts

(B)

1. _____ Occurs when the subjects are unaware of the researcher; they do not know about the research study and proceed with their activities in a normal manner A. Obtrusive research B. Unobtrusive research C. Participant observation D. Elite interviewing

(B)

1. Inductive approaches move from ____ to ____ A. theory to data B. data to theory

(B) - you start with the data and work up to theory, deduction is opposite

1. "Democracy is the best type of political regime for human societies" This is an example of __________ A. An empirical statement B. A normative statement C. A theoretical statement D. A hypothesis

(B) A normative statement

9. In sampling, the group we wish to generalize about is known as the _____ A. Sample B. Population C. Case D. Parameter

(B) Population Sample== they're the participants you use)

1. Operationalization is ____ A. A concept B. The movement from variable to concept C. The movement from concept to variable D. A variable

(C)

1. The _____ approach seeks to understand political life through the study of a large number of cases A. Qualitative B. Determinant C. Normative D. Quantitative

(D)

Steps in Index Construction

1. Establish the many dimensions of the concept in question 2. Develop measures for each dimension 3. Pre-test the index 4. Pilot test to assess the consistency of the items in the index

Five Types of Surveys

1. Mailed 2. Internet 3. Individual or group administered 4. Interviews 5. Phone • Some researchers may employ more than one (i.e. multiple methods) type of survey in a research project as Jackson and Verberg suggest (page 105): "They may begin with a mailed survey, and follow-up with a telephone survey to encourage those who have not completed the survey with the option of doing so on the telephone."

The Ten Commandments of Interviewing

1. Never begin an interview cold 2. Remember your purpose 3. Present a natural front 4. Demonstrate aware hearing 5. Think about appearance 6. Interview in a comfortable place 7. Don't be satisfied with one-word answers 8. Be respectful 9. Practice, practice and practice some more 10. Be cordial and appreciative

Bias at Each Stage of the Research

1. Selection of the research question 2. Sampling design 3. Funding 4. Data collection a. Researcher affect b. Socially desired bias 5. Data analysis a. Coding errors b. Data massaging: The practice of playing with the data until the analysis provides your expected outcome 6. Reporting of findings 7. Objectivity versus advocacy

Content Analysis: Things to Consider

1. What is your unit of analysis? Is it a song? A book? Letters to the editor? Sentences? 2. How will you select your sample? Remember, standard sampling techniques try to reduce bias from the choice of the materials selected. As Jackson and Verberg (2006: 131) note, "Since the analyst typically wishes to describe some phenomenon accurately, it is important that the procedures used represent all instances of the phenomenon fairly" 3. What is being measured? A word? A theme? Once we decide what is being measured, we have to build our codes and definitions for those codes (I.e. type of film like horror, action-thriller, drama, comedy). 4. Before coding, a tally sheet should be constructed or you may wish to use computer software programs that can retrieve text.

Quantitative Approach: Advantages vs. Disadvantages

Advantages Best for providing in-depth understanding of phenomena or experiences. Greater validity, richer understanding. Higher quality responses. Disadvantages Worst for providing generalizations. Harder to achieve objectivity Know more and more about less and less Time consuming

Example of Quasi-Experimental Design

Example of Quasi-Experimental Design • Example. Loewen, Koop, Settle and Fowler (2012) employed a natural experiment to determine whether proposing legislation in the House of Commons affects electoral outcomes (i.e. do MPs who propose legislation get rewarded with more votes) The right of backbench MPs to propose legislation is determined through a lottery by the Speaker of the Commons. This experiment employed data from the legislative and electoral records of backbench incumbents from two sessions: 2004-2006 (38th Canadian Parliament) and 2006-2008 (39th Canadian Parliament). • The experimental group consisted of backbench MPs who won the lottery to propose bills or motions. In both Parliamentary sessions, this group consisted of 165 MPs (note, Cabinet members excluded). • The control group consisted of MPs who did not win the lottery. In both Parliamentary sessions, this group consisted of 239 MPs. • Hypothesis: Backbench MPs from the government who propose legislation will receive an "electoral boost from the chance to propose and debate legislation, but opposition members do not". • The results from this experiment support the hypothesis statement o The ability to introduce legislation increases the share of votes for government candidates (2.7%) or the equivalent of a 7% increase in the probability of winning the next election.

Why can't we make general sweeping statements with the Classic Experimental Design?

It can't be generalized because of the small sample size, but randomization would be used to reduce that bias

Methodological Differences between Quantitative vs. Qualitative

Quantitative Data format: numerical Data reduction: variables Interpretation of data: one inherent meaning Data processing and reporting: math, stats, graphs Probability Qualitative Data format: symbols, words, images Data reduction: emergent themes Interpretation of data: several or many meanings Data processing and reporting: conceptual and verbal Plausibility

Epistemological Differences between Quantitative vs. Qualitative

Quantitative Primary intent: test hypothesis Ultimate goal: generalizability (large sample size) Theory development: deductive Positivist Qualitative Primary intent: address questions Ultimate goal: Specificity (small sample size) Theory development: inductive Interpretivist

Who doesn't receive the stimuli?

The control group

Levels of Measurement

Three levels of measurement: nominal, ordinal and ratio.

Test Yourself: List the Unit of Measurement for Each

Variable: -Age -Ideological Position (i.e. Liberalism) -Regime Type (Auth., Dem., Totalitarian) -Population -Unemployment rates

Exercise: Identify the Independent/Dependent Variables • Political conservatism/support for balanced budgets • Support for national child care program/gender • Completion of a government-sponsored job training program/full employment • Political party membership/age • Liberalism/support for social welfare programs

o Political conservatism (X) /support for balanced budgets (Y) (could be reversed) + o Support for national child care program (Y) /gender (X) (can't be reversed, gender can't change) o Completion of a government-sponsored job training program (X) /full employment (Y) + o Political party membership (Y) /age (X) (can't be reversed, age can't change) o Liberalism (X) /support for social welfare programs (Y) (could be reversed) +

Exercise: Develop a Null and Alternative Hypothesis for... • Political conservatism/support for balanced budgets • Support for national child care program/gender • Completion of a government-sponsored job training program/full employment • Political party membership/age • Liberalism/support for social welfare programs

o Political conservatism/support for balanced budgets ♣ WRONG: because it says higher rates aren't associated with higher rates, b/c what about lower rates? Just say no relationship • i.e. NULL: Higher rates of political conservatism is not associated with higher levels of support for balanced budgets • i.e. ALT: The more politically conservative an individual is the more likely they are to support balanced budgets o Support for national child care program/gender ♣ RIGHT: because there's directionality to it from the word "more" • i.e. NULL: There's no relationship between gender and support for national child care programs • i.e. ALT: Women are more likely to support national child care programs • Could also do that the relationship between men and support for national child care programs is unclear o Completion of a government-sponsored job training program/full employment ♣ i.e. NULL: There is no relationship between completing a government sponsored job training program and full employment ♣ i.e. ALT: Completion of a gov. sponsored job training program increases the likelihood of gaining full employment o Political party membership/age ♣ ??? ♣ ...make sure it's clear i.e. as age increases, political party membership increases ♣ Syntax error: the older one is the more likely they will become a political party member ♣ Stick to one or the other. Either they or one o Liberalism/support for social welfare programs ♣ ???

How Do You Record the Data for Focus Groups?

o Taking notes and transcribing them later o Taping the discussion (I.e. audio, video) which are transcribed later (though you need to let participants know this). • Regardless of which approach you choose, you need to think about confidentiality (I.e. who will have access to the video?). You may want to have all participants including the moderator to sign a confidentiality agreement that anything that is said in the group stays in the group.

A Difference to Note (Direct Method of Gaining Knowledge)

• Adopting a qualitative approach is a direct method of gaining knowledge. This is in comparison to non-reactive studies that involve indirect data collection such that the person being studied has no opportunity to react to the observations. • Non-reactive studies are also referred to as unobtrusive research and the three main types of this research method are: 1. Content analysis 2. Secondary data analysis 3. Historical/comparative analysis (note, generally relies on secondary data analysis)

Advantages and Limitations of Classic Experimental Design

• Advantages o Can provide convincing evidence of causation o When the experiment is double-blind, there is control over bias • Limitations • Experiments occur in a contrived setting and the concern is that the experiment may have some effect on the results obtained. In the 'real world', the results may differ because people know they are involved in an experiment • Not all topics can be studied through experiments (I.e. studying the effectiveness of a new birth control pill whereby you would have to deny birth control to the women in the control group - ethical concern.

Content Analysis: Advantages and Disadvantages

• Advantages: o Cost-effective (especially compared to a survey) o Unobtrusive o Allows for a study over a period of time to identify trends in society (I.e. portrayal of women in media from 1900 to 2000). • Disadvantages: • Is limited to examining only those messages that have already been recorded • Not effective for testing cause-effect relationships between variables

Challenges of the Historical/Comparative Analysis

• Challenges: o concepts may not be universally agreed upon (I.e. incest may mean different things in different cultures). o Data collection procedures may differ (I.e. the unemployment rate in country X may be collected differently than the unemployment rate in country Y) o Translation / language may be a concern

Seeing Things

• Concepts exist at the level of theory (I.e. there is no true measure for grade improvement) but rather it is how we establish measures for this concept. There is NO right way of seeing things but there are BETTER ways where validity and reliability are a must

Content Analysis

• Content analysis: a technique for making inferences by objectively and systematically coding content in the messages conveyed by the media (radio, TV, film) or written materials (plays, songs, novels, newspapers). • Example: Going through Canadian national newspapers to measure how much print space is written about the issue of domestic violence. • Example: A sociologist analyzing gender stereotypes in the books that school children read

Convenience and Snowball Sampling

• Convenience sampling refers to a sample that is drawn from individuals that happen to be available (I.e. going to a shopping centre and interviewing shoppers or the person walking down a street that is asked to be polled). • Snowball sampling is a means of selecting people to interview when a population list is unavailable. Start by interviewing some individuals that fit the criteria of the study and ask them if they know anyone else to interview who also fits the criteria.

How Do We Build Theories: Deduction and Induction

• Deductive reasoning (or deduction): you start by getting to the core principles/beliefs and then the entire research project builds on those principles o Example: if you believe humans are selfish then you execute the research project built around this belief • Inductive reasoning (or Induction): where you look at the evidence first (i.e. what you see/observe) and then figure out what the core principles are working backwards o To better understand these two forms of reasoning, we need to understand the three elements of the scientific model of social research: theory, conceptualization and operationalization o Is deductive/inductive better than the other? Hard to answer as each adds different things

Experiments

• Experiments are attempts to recreate reality so as to determine causality, but within a controlled environment • The goal of an experiment is to determine how one factor shapes another factor, independent of all other factors (all things being equal). • We experiment every day How can I change my route to work to save time? • Experiments work best for explanatory research rather than descriptive research. • Three conditions are necessary for establishing causality: o Temporal order (I.e. X has to happen before Y) o Consistent pattern of association (You want it to happen again and again to establish a pattern) o Elimination of plausible alternative causes

How Do Focus Groups Work?

• Focus groups work best with 6-12 participants who agree to meet at a pre-determined location. A facilitator (I.e. perhaps yourself or someone you hire via a research firm) raises certain issues and attempts to guide discussion but does not generally have a list of questions written in stone. • The facilitator uses a 'moderator's guide' to facilitate the focus group. The moderator's guide must consider the level of language for the focus group (I.e. language used by 18 year olds is different from the language used by 65-year-old participants). • The moderator's guide provides an outline and the sequence for what the moderator should say and do next. It should include: o Introductions of the participants (I.e. introductory activities) o Statement of the basic rules or guidelines (I.e. we will be meeting for 2 hours, there is coffee available, etc.) o Short question and answer discussions o Special activities or exercises (I.e. here is a copy of MLC's draft brochure on gambling addictions, with your highlighter please highlight any text that you find confusing or awkward).

Sample Size (Quantitative)

• How many people should you talk to if you want to say something about the population? If you want a representative sample then it must be a random sample to provide results that deviate only slightly from the population's characteristics. • Sample size is dependent on: confidence level and sampling error o Confidence level is an indicator of the degree of accuracy for a sample. We express the accuracy of our sample statistics in terms of level of confidence that the statistics will fall within. For example, you commonly hear that the results form a study are accurate 19 times out of 20 (which is mathematically equal to a 95% confidence level).

Qualitative Research: Sample Size

• If representativeness is undesirable, a large sample size and random selection are not required. Qualitative research is not concerned with making broad predictive statements about the population. Rather the focus rests with gaining richness of understanding about the lived experience. • So, how do know what sample size to select in more qualitative research? o Saturation

Internal Validity

• Internal Validity: does the independent variable really cause the dependent to change or is the change the result of something else?

Why Use Focus Groups?

• Jackson and Verberg argue that they provide a dynamic where the individuals in the group can learn from each other and build off of each other's ideas. The researcher can spot consensus and a strong advantage is that the responses of individuals may lead to important issues the researcher had no idea about.

Ethics

• Key ethical question: Is it ethical to deceive people you are studying in the hope that they will confide in you as they may not have confided in you if they knew you were a researcher? • Related to this ethical question is a scientific one according to Earl Babbie. Deception is done on the belief that answers will be more valid and reliable (I.e. subjects are more honest, natural). If people know they are being studied, they may: 1. Expel the researcher; 2. Modify their behaviour to appear more respectable 3. The social process may change completely (I.e. a radical group may curb use of violence if it knows that a researcher is there watching). • BUT o If you are a complete participant, you must participate and thus affect the social process of the group you are studying. Anything the participant observer does or does not do will have some effect on what is being observed; this is inevitable but sensitivity may reduce this effect. ♣ Example: the group asks for your ideas about staging an anti-globalization protest against the WTO. No matter what you say or don't say - you affect the social process.

Classical Experimental Design

• Key terms: o Independent variable - experimental stimulus - present or absent - X. o Dependent variable (the effect - Y) o Pre-test and post-test (measures dependent variable before and after stimulus to determine change due to stimulus) o Experimental versus control group: ♣ The experimental group receives the stimulus ♣ The control group does not receive the introduced stimulus but will receive a placebo or alternative treatment. • The level of the dependent variable is tested for the experimental group (that receives the introduced stimulus) and the control group before and after exposure to the treatment (I.e. the independent variable). • Only the experimental group is exposed to the stimulus. The control group gets a placebo or alternative treatment in order to control for external changes due to factors other than the treatment • Selection of sample for experiments can employ randomization, but normally the sample is relatively small. As such, experimental results cannot likely be generalized even if sample is randomly selected because of the small sample size • May want to conduct a double-blind experiment: Neither the subjects nor the researchers are aware of who is in the control group versus the experimental group to avoid researcher bias

Measurement

• Measurement links the conceptual and operational stages. • Remember, as we move down from theory to concepts to measures, we become more and more precise. No longer abstract, but concrete.

Data Collection Methods in Qualitative Studies: In-Depth Interviews

• More akin to an "interview schedule" and the goal is to record the activities of individuals and any possible explanations they offer about their world. Usually tape-recorded to preserve verbatim descriptions provided by interviewees. Quotations from the interviewees are contained in the final report to illustrate certain points. • Often used when the target group is high-status (I.e. political leaders) as they are less likely to respond to a highly-structured questionnaire

Concerns with Internet Surveys

• Multiple Responses: This occurs with internet surveys because there is a lack of control over the respondents and the number of times they respond. Researchers may ask for specific birthdate repeated in the survey to catch multiple respondents or search submissions for identical responses. • Sample representativeness: occurs because it is very challenging to identify or access the population in which to randomly draw the sample and ensure that the sample represents the population in key areas (i.e. gender, age, income, etc.). Researchers weight the data on the backend to represent the population.

Degrees of Participation

• Not all participant observers will participate in what they are studying though they study it directly at the scene of the action. Thus, there are degrees of participation in the daily life of the group s/he is studying. At one extreme is full participation while the other extreme is no participation (I.e. just observation) in social action. These degrees form a continuum of participation

How to Calculate Sample Size (Quantitative)

• One way is to do the math yourself or use an online calculator or better ask a statistician to assist you: • Example: https://www.surveymonkey.com/mp/sample-size-calculator/

Question Types for Survey Questionnaires

• Open-ended questions are designed so the respondent gives their own answer to the question. Try to place these at end of survey. • Example: What do you feel is the most pressing issue facing Canada? ____________________________________________________ • Close-ended questions force the respondent to choose from a list of possible answers supplied by the researcher. Or, can elicit a single word response like 'yes' or 'no'. • Example: Which of the following issues is the most pressing for Canada? (Select one only please) o Health o Taxation o Education o Don't know o Other, please specify:____________________________

Moving to Operationalization Stage...

• Operationalization is one step beyond conceptualization and is the process of developing operational definitions or specifying the exact operations involved in measuring a variable.

Two Types of Sampling Methods

• Probability or random sample: the sample is selected in a manner that attempts to ensure that all individuals have an equal chance of being selected. Randomization is necessary for using the sample to draw conclusions about the population. • Nonprobability or nonrandom sample: the sample selected without regard for its representativeness

Quantitative and Qualitative Approaches

• Quantitative research is linear and follows a series of steps: o Define the research problem o Conduct a literature review o Formulate hypothesis statements o Operationalize definitions o Design instrument for data collection o Obtain research ethics approval o Collect data / analyze data o Interpret data and draw conclusions • Qualitative research is non-linear: IMAGE 7

Quasi-Experimental Design

• Quasi-experimental design come as close as possible to experimental design in order to measure the impact of a treatment. A quasi-experimental design is one in which a design has not been possible to do any or all of the following: o randomly assign subjects to a treatment or control group; o control the timing or nature of the treatment. • They attempt to introduce elements of the experimental design, but within a field setting. • They are used because often you cannot select a control and experimental group. These are used to determine policy impacts when there can be no control group. o Example, this design may be used to investigate welfare payments as disincentives to seeking work. This topic could not be conducted using a classic experimental design because welfare payments cannot be denied to a group.

Sampling Size: Sampling Error (Quantitative)

• Sampling error: the difference to which a sample differs from the population usually expressed as '1%' or '2%' or '3%' etc. This is the error that a researcher is willing to accept and is expressed in percentage form. • A lower tolerance for error (i.e. 1% or 2%) requires a larger sample size. However, the law of diminishing returns applies to sample size. At a certain point, the gains made in reducing the error diminish as sample size increases. o For example, to move from a 2% error to a 1% error rate requires an increase in the sample size of 7,500. If precision is critical (i.e. predicting electoral outcomes) then a larger sample size may be desirable. o Conversely, a higher tolerance for error (i.e. 7%) requires a smaller sample size. • If you wanted to be quite confident about your results, then it is best to adopt a 95% confidence level (this is the standard level) and you may also want to adopt a lower error of perhaps 2%. Thus, if you adopt these figures, you can say that you are 95% confident that your survey results will be with a plus/minus 2% range. • If one of your survey results was that 44% of women vote NDP then you are 95% confident (or 19 times out of twenty) that 44% of the women in your population will vote NDP plus or minus 2% (so, 42% and 46%). There is a 5% chance (100% minus 95%) that this result would not hold, but the odds are small.

Sampling Disasters... President Alf Landon

• The Literary Digest, based on its polling, predicted that Landon would beat Roosevelt in the 1936 election. The magazine based its prediction on mailings to about 10 million Americans for a total sample of 2 million. • With such a large sample, why was the magazine incorrect? o George Gallop pointed out that the mailing lists were derived from the phone book and lists of automobile owners. The sampling frame was not appropriate because the lists overrepresented those individuals with higher SES, who would be more inclined to vote for the Republican (Landon). Given that this election took place during the Depression, there were more poor Americans who would be more inclined to vote for the Democratic candidate (Roosevelt). The sampling frame did not accurately reflect the characteristics of American voters.

What is the key to randomization?

• The key to randomization is that the list itself is random - periodicity is bias introduced as a result of the list itself. The list must be random. Alphabetical lists may not be random (I.e. list of people by last names may have some common names over-represented like 'Smith').

Secondary Data Analysis

• This involves the analysis of data that has already been collected for some other purpose. For example, Stat Canada collects data on rates of volunteerism. Or, think about economists who use unemployment statistics. • This is a cost-effective research method and you can test causal models. • Must be cautious whenever you work with survey data because there may be concerns with validity.

Data Collection Methods in Qualitative Studies: Participant Observation

• This is a method where the researcher becomes involved intensely of some culture, community or organization for an extended period of time. You may gain entry by being honest about your role as a researcher (overt participants) or you may have to pretend to be a participant (covert participant; Laud Humphrey's was a covert participant) • Example: an anthropologist who lives with a pre-literate tribe who becomes a member of the tribe to experience everything that they are experiencing.

Quota Sampling

• This is a type of non-probability sampling. This involves sampling until you get enough individuals that have certain characteristics in your sample (I.e. 50% men and 50% women). Thus, while you may have specific needs for respondents (I.e. gender, age, religion) you take people who fill your quota and these needs. o Example: going knocking on homes until your sample size is complete. • Although quota sampling resembles probability sampling, it has several problems: 1. It is non-random because you select cases on a first come, first served basis which can introduce bias where you only speak with people who are easiest to find people or most willing (e.g. you may avoid speaking with people living on the 10th floor of an apartment block with no elevator); 2. There is no way of determining the accuracy of the sample.

Historical/Comparative Analysis

• This is research that typically involves cross-cultural or historical analysis of social behaviour. Note: Jackson and Verberg (2007: 134) note, "Comparative research is not really a special type of research with its own methodology; it is the research process itself". o Example: Historians comparing American and Canadian history. o Example: Policy analysts comparing the social welfare regimes of Canada, Sweden and Germany.

Multi-Stage Area (Cluster) Sample

• This type of sampling is used when a list of the population (I.e. the sampling frame) is difficult to obtain. • Key here is to select randomly at each stage o Example: want to sample Canadians that attend Roman Catholic Churches but say a list of these people did not exist. Start with picking a random sample of dioceses then from the diocese randomly pick churches from that diocese. Then speak with the Priest about sampling his congregation. o Procedure then is to randomly select a set of groups from which all or a random sample of individuals will be selected

Stratified Random Sample

• Thus far we have discussed two types of probability sampling techniques: simple random and systematic random. The stratified random sample is NOT an alternative to these methods but is a modification of their uses. Simple random and systematic random sampling techniques ensure a degree of representativeness. What stratified does is to ensure a greater degree of representativeness • It's simply an extra step • Rather that creating a sample from the population at large, you ensure that samples are drawn from subsets within the total population. To get a stratified sample of university students, you need to organize the population (e.g. if you choose year in university as the stratifying criteria, then you draw a sample from the freshman year, and another sample from the sophomore year, and another sample from the junior year and another sample from the senior year). • In a nutshell, this sampling technique allows for greater represenativeness because you have ensured that they have been included in the sample. How do employ this technique? In three steps... o Step 1: select the stratifying variable (I.e. ethnic background) o Step 2: determine your sample size from each stratified group and develop a sampling frame for each category o Step 3: use a random sampling procedure to select from within each group. • Example: Let's say your research project wanted to avoid distortions due to the chance of over representation of particular ethnicity in the final sample we would stratify the sample by ethnic background to guarantee that each ethnic group will be represented in the sample in its correct proportion. IMAGE 6

Data Collection Methods in Qualitative Studies: Focus Groups

• Typically consists of 6-12 individuals who form a group to discuss a topic suggested by the facilitator. This technique is heavily used by market researchers who ask consumers to evaluate a product for example. A social science example is Gamson's (1992) use of focus groups to examine how Americans frame their views on certain political issues like affirmative action.