SCIE211 Intro to Scientific Analysis & Reasoning

Occam's Razor

"entities must not be multiplied beyond necessity" or we should consider the less accepted of more complicated theory only if the more commonly accepted (or simpler) theory cannot adequately explain the phenomenon.

Describe the role of rationalism in science.

'Think rationally,' basically means to think clearly, using reason or logic. So, rationalism is the view that regards human reason as the primary means of discovering knowledge and determining what is true or false. ... Sometimes rationalism is called scientific rationalism, but it means basically the same thing.

Explain how critical thinking may be applied to the most common type of statistical information: polls.

Assess evidence provided by the polls; identify the sampling method used, the methodology and questions, and non-response information. Also consider the political and social context in which the poll was done. Consider other polls that may conflict.

Discuss what statistical sampling is and its difficulties.

Because of the cost of sampling, pollsters have come to accept the plus or minus 3 percentage points level of accuracy as standard for national polls. The main challenge in real polling is asking appropriate questions and actually getting a random and unbiased sample. Others: refusal to be interviewed (non-response), question wording and question order, interviewer bias, weighting by demographic control data

Explain how critical thinking facilitates the acquisition of scientific knowledge.

Critical thinking facilitates the acquisition of scientific knowledge by teaching people to always think about the why and how of a situation or idea.

Explain the problem of the blind men and the elephant.

It is a story of a group of blind men who have never come across an elephant before and who learn and conceptualize what the elephant is like by touching it. Each blind man feels a different part of the elephant's body, but only one part, such as the side or the tusk. They then describe the elephant based on their limited experience and their descriptions of the elephant are different from each other. In some versions, they come to suspect that the other person is dishonest and they come to blows. The moral of the parable is that humans have a tendency to claim absolute truth based on their limited, subjective experience as they ignore other people's limited, subjective experiences which may be equally true.

Identify the key parts of a research article.

Objective, Design, Subjects, Intervention, Main outcome measures, Results

Identify the basic types of graphs.

Pie, line, bar

Identify extraneous or confounding variables.

The mixing of factors that obscures the actual causal factor

Explain what opinion polls are and some of the problems with polling.

The same question or questions are given to a sample and the findings are reported. Some issues with polling include question bias: the phrasing of the questions can influence the answers; misrepresenting the sample as a population

Explain how we determine support for causation.

i. There is an appropriate temporal relation - causes (in general) must precede their effects ii. There is an appropriate dose relation - more cause, more effect iii. There is a scientifically credible explanatory model consistent with the data iv. The claim fits with other scientific understanding v. The claim is consistent with relevant experiments, for example, on animal models or in Petri dishes

What are the four basic CRITICAL QUESTIONS?

i. What is being claimed? ii. How good is the evidence? iii. What other information is relevant? iv. Are relevant fallacies avoided?

Outliers

values far from the others

Ad hominem fallacy

when speakers attack the person making the argument and not the argument itself

Gambler's fallacy

1) to assume that because probability is long-term frequency, there is a natural tendency (or some gamblers) to think that short-run deviations from the likely long run will be "corrected" by chance. 2) Thinking that the previous events will be repeated because the dice are "hot"

Common ways graphs can be mis-leading:

1. Scale not starting at zero. 2. Scale made very small to make the graph look very big. 3. Scale values or labels missing from the graph. 4. Incorrect scale placed on the graph. 5. Pieces of a Pie Chart are not the correct sizes. 6. Oversized volumes of objects that are too big for the vertical scale differences they represent. 7. Size of images used in Pictographs being different for the different categories being graphed. 8. Graph being a non-standard size of shape.

bandwagoning fallacy

accepting a statement as true because it is popular

fallacy of anecdotal evidence

An unwarranted generalization from one's own experience, or from someone else's

Explain the danger of confusing correlation with causation.

Causation is the relationship between cause and effect. So, when a cause results in an effect, that's a causation. ... When we say that correlation does not imply cause, we mean that just because you can see a connection or a mutual relationship between two variables, it doesn't necessarily mean that one causes the other.

Explain what a fallacy is and give an example.

Common arguments that seem persuasive but do not provide adequate support for their conclusion. 1. Hasty conclusion - coming to a conclusion with too little investigation and inadequate evidence. 2. Biased sampling - not using an unbiased and representative sample of the population being studied

Define an observational study.

In observational studies the researchers collect data about things; in case-control studies, two groups are compared, a cohort study is prospective

Identify and explain possible problems with experimental studies.

Selection Bias: To avoid selection bias in an experimental study, the subjects must be randomly assigned to two different groups (randomized controlled trials or RCT) There is also the potential for bias in assessing outcomes; to prevent this, a double-blind study should be used (subjects and researcher are unaware who is in which group)

Explain the benefits and importance of the study of statistics.

Statistical knowledge helps you use the proper methods to collect the data, employ the correct analyses, and effectively present the results. Statistics is a crucial process behind how we make discoveries in science, make decisions based on data, and make predictions.

Describe what a correlation is and how it is established.

That something is accompanied by something else - whether it's actually true or not. A correlation exists when two factors vary together (co-vary). There are two different ways of talking about correlations: categorical and continuous.

Explain the role of experimental methods in testing cause and effect relationships.

The experimental method is the best way to control for confounding factors, because in an experiment the researcher manipulates the causal factor being tested and controls all other variables.

Evaluate thinking in regard to causal claims about issues in science.

The ideal method for establishing a causal relationship is the randomized, double-blind experiment, because such an experimental approach can eliminate all explanatory factors except the cause being tested. Establishing most claims requires repeated testing and peer review

to have the ability and inclination to assess claims and arguments by evaluating evidence using appropriate criteria. Knowing how to evaluate and what questions to ask can reduce our chances of being deceived and can be free to make intelligent use of credible statistical information.

To think critically is to have the ability and inclination to assess claims and arguments by evaluating evidence using appropriate criteria. Knowing how to evaluate and what questions to ask can reduce our chances of being deceived and can be free to make intelligent use of credible statistical information.

Explain the meaning of "correlation does not imply causation."

We can only know if being in a particular category is correlated with some possible effect by comparing the rates of the effect in those who are in the target category with those who are not.

Random sample

an unbiased sample; one in which every member of the population has an equal chance of being selected.

Post hoc ergo propter hoc fallacy

from the Latin for "after this, because of this;" assumes that because one event happened after another, then the preceding event caused the event that followed

Confidence level

how likely it is that the true answer is within the margin of error of the sample.

Explain how to interpret mainstream research.

i. Ask: 1. What is the actual claim in the study? 2. How good is the evidence? 3. What is the context? 4. Any fallacies? ii. Get the big picture - dig into other references, find systematic reviews of research literature iii. Meta-analysis - bring together the data from a multitude of studies, collect data into one big sample and then do statistical analysis on this new "sample" iv. Look for the "desk drawer fallacy" v. Look for contradictory evidence Evaluate based on the weight of evidence using the four-level hierarchy (negative, inconclusive, weakly positive, strongly positive)

Explain why people shouldn't trust observational studies.

i. Confounding factors can plague observational studies ii. Observational studies can be biased iii. Experiments are more credible than observational studies

Explain and illustrate some of the fallacies of data quality and characterization.

i. Data can be made up numbers or guesstimates "mythical numbers" (hard numbers vs. soft numbers) ii. False precision: using a level of precision (e.g., a number of decimal points) that cannot be justified by the accuracy of the original data iii. Change in reported incidence vs. actual change: From crime statistics to health data, changes in definitions or awareness can create the appearance of real changes when actually nothing has changed but the reporting iv. Misleading aggregation: Combining a wide variety of information in one statistic, for example, crime statistics that lump together everything from disturbing the peace to homicide

Explain how we gain scientific knowledge.

i. Make an observation or observations. ii. Ask questions about the observations and gather information. iii. Form a hypothesis — a tentative description of what's been observed, and make predictions based on that hypothesis. iv. Test the hypothesis and predictions in an experiment that can be reproduced.

Explain the advantages and disadvantages of observational studies.

i. Retrospective studies may rely on samples of convenience rather than truly random samples ii. Prospective studies can reduce many of the uncertainties involved in retrospective studies iii. In prospective studies, sample groups are identified by the input or suspected contributing factor and not the outcome

Explain the main types of observational studies

i. Retrospective: look backwards. Identify a group of people and attempt to find factors that are different in this group compared to a group of similar people (control group) ii. Prospective: forward-looking. Isolate a sample group, or cohort, and follow the subjects over time, noting differences between them

Explain the importance of reproducible experiments and the peer review process to help overcome the problems of experimental studies.

i. The burden of proof is on the proponent when making a scientific claim. ii. Peer Review process is where experts in the field examine and evaluate new research. iii. If research turns up controversial or surprising results it must be replicated. iv. The process by which successful theories are supported by numerous studies is often called "convergence"

Explain the problems that influence the quality and reliability of polling information.

i. When evaluating the evidence that the poll sample provides, we need to look for two sources of bias: 1) bias in the sampling process, e.g. a sample favoring a certain group called selection bias; and 2) biases resulting from the interview process, e.g., loaded questions, respondent lying, non-response, etc. ii. Self-selection bias: the bias that results from allowing subjects to choose whether or not to respond. iii. Respondent selection bias: tendency of people to deceive pollsters iv. Question order bias: the order in which questions are asked can influence the answer (example: presidential approval ratings tend to be lower when asked at the end of a survey where other issues were brought up first). v. Introductory context bias: the introduction of a survey contains biased language that could influence responses vi. Sponsor bias: manipulation occurs when the sponsor of the poll has an interest in the poll outcome

The fallacies of "average"

ignoring the spread - sometimes the median is more representative; the paradoxical average - when the average is so misleading that it actually produces a paradox, known as Simpson's Paradox.

Stratified sample

produced by dividing the population into large groups, often by geography, and then randomly picking subjects within each of these groupings.

Statistical significance

the apparent difference is not just the result of the natural variation involved in sampling.

Hume's Rule

the more surprising the claim, the more evidence we should require before accepting it.

Representative sample

the people in the sample should be like those in the population.

Desk drawer fallacy

the problem whereby many studies that fail to find a correlation simply don't get published, which can create an illusion of scientific convergence.

Margin of error

the range of percentage points around the sample percentage where the true population average is likely.