Dissecting the research article

Two basic evaluation questions regarding the materials used

1. Was there adequate selection and measurement of the *independent variable*? 2. Was there adequate selection and measurement of the *dependent variable*?

Sample

A subset of the population that is included in the study ex. 10 adults with Down Syndrome recruited in Pittsburgh, PA

Rationale sub-grouping (cluster sampling)

Instead of randomly selecting individuals... - units (groups or clusters) of individuals are identified - a random sample of units is then selected - all individuals in each unit are assigned to one of the treatment conditions - we see this often in classroom study *Units must be HOMOGENEOUS in order to avoid bias*

Data analysis

Usually last section of the Methods Describes how the data will be organized, summarized and assessed statistically This section links the methods section to the Results section

Research hypothesis

Brief and to the point (more focused) Testable

Abstract

Brief summary of the research article Designed to allow readers to get the general idea of the content of the article Typically includes: - concise rationale for why the study was conducted/purpose - short explanation of methodology used - key results - primary implications or conclusions

Ideally, rational in the introduction should:

- Stem from the general statement of the problem - Presents the case for studying selected aspects of the problem, and may identify limitations

Four key parts of the abstract

1. Rationale/purpose 2. Method 3. Key results 4. Implications

By the end of the introduction, you should be able to:

1. Identify the overall purpose of the study 2. Identify the research questions and hypotheses 3. Explain why the study is important

Scales of measurement

1. Nominal 2. Ordinal 3. Interval 4. Ratio

In communication sciences and disorders (CSD), we classify measures as either ___________ or _________

*Observer measures* *Instrumental measures*

Introduction provides:

1. *Critical synthesis* of the background information needed to understand the study 2. *Explanation* of why the study is important: identification of a problem 3. *Overall purpose of the study* 4. *Direction* for what the authors expect to find (or hypothesize) Overall: what problem are they trying to address and why?

By the end of the method section, you should be able to:

1. *Understand* who the participants were and what methods were used in the study 2. *Replicate* the study if you wanted 3. *Identify strengths and weaknesses* of the design used

Choosing a simple random sample

1. Define the population 2. List all members of the population 3. Assign numbers to each member of the population 4. Use criterion to select a sample

Exclusion criteria

Attributes of participants that render them ineligible to participate Responses of participants that require their removal from the study

Evaluating the method with one question

Are the measures selected to represent or capture the dependent an independent variables *reliable* and *valid*?

Questions to ask when deciding whether the article is worth reading

Are the methods reasonable? Are the conclusions interesting/applicable to the problem you have identified?

Inclusion criteria

Attributes of participants that are essential for their selection to participate in the study

Observer measures (of behavioral variables)

Behavioral observation is used for measurement of language varables, like MLU, or speech variables, like intelligibility, naturalness, or fluency

Why is the scale of measurement important?

Dictates the design and statistical analysis - accurate quantification of communication variables can be achieved ONLY through careful measurement procedures that are designed to yield *reliable* and *valid* results

Questions to ask when reading the method section

Do the methods match the original problem logically? Does the method make sense for the research questions? - Participants? - Setting? - Materials? - Design? - Procedures? Are there potential weaknesses in the methods? Would there have been a better way to address the overall problem?

Questions to ask when identifying the relevance of the article

Does it seem to be related to the problem you have identified?

Instrumental measures (of physical variables)

Electronic insturmentation is used to measure physiological variables like airflow, or acoustic variables, like vowel formants, voice onset time

Methods section provides:

Explanation of the *method* employed to address the problem Details about participants, measures, and procedures Overview of study design and the approach to the analyses Rationale for statistical analyses

When critically evaluating a methods section...

Focus on identifying factors related to the participants, materials or procedures that could have confounded the results Factors related to the materials, tasks and protocol can affect the quality of measurements, thereby posing a threat to the validity of the results

Populations and samples are important because:

Helps us understand and evaluate why the researchers chose their participants

Importance of sampling and selection criteria

How researchers sample and selects participants *impacts the generalizability of the findings* Biased sampling can impact the *validity* of the results "Sampling bias" = sampled data does not represent the population accurately (threatens validity)

Questions to ask when reading the introduction

Is the argument for the research logical? What is the overall purpose? Is it clearly stated? Is the rationale for the study easy to understand? What is the background information you have gained? Is it supported by clear examples and *citations*? *If you have a large number of unanswered questions, there may be a problem*

Convenience sampling

Most common Collecting the most convenient sample Should include some attention to sample characteristics True random sampling is not always feasible in most cases - important to critically evaluate the criteria used to select participants from a convenience sample

Terminology: Participants versus subjects?

Participants implies *consent* and *mutual understanding* - in our research, participants are usually humans

The method section of a paper includes:

Participants/subjects Data acquisition (materials and procedures) Data analysis plan

Random sampling

Randomly selecting the research sample from the larger population Very difficult to do unless the population of interest is *very small* Each member of the population has an *equal and independent* chance of being chosen *The sample should be REPRESENTATIVE* of the population

Selection criteria

Refer to the attributes within a sample of participants that are *desirable or undesirable* for addressing a specific research question regarding a specific population Remember: researchers should describe in detail their choices for selecting a sample

Generalizability

Refers to the level of applicability of research finding to a different group of people Research findings *should* be generalizable to groups of people that *were not directly included in the study* You cannot know that findings will generalize

(Sample or population?) 30 students enrolled in Research in SLP

Sample

Research protocol

Sequence of tasks performed by the subjects Manipulations of the independent variable Measurement of changes in the dependent variable

Research question

Stated as a question May ask if something exists (description), if there is a relationship, or a difference Follows from the rationale Can be broad

Population

The larger group of people researchers are interested in studying ex. Adults with Down Syndrome living in the US

Method

The researcher describes what is *done* to the participants with the *materials* The procedures serve as a guide to replication

Parts of the article (which directly map onto the scientific method)

Title Abstract Introduction/Literature review - Identify the problem Method - determine the method to investigate the problem Results - Present the results of the investigation Discussion - form a conclusion References

What to look for in the participant section

Were the participants recruited *retrospectively* or *prospectively* Sample size Selection (inclusion/exclusion) criteria for participants Rationale for participants included Demographic information - male-to-female ratio - age - SES - Race/ethnicity - Time post-onset

Method materials

Where the researcher identifies the *materials* used to *measure* the variables being investigated

If your sample is *representative*, then what can you guess?

You can guess what individuals *outside of the study would show a similar pattern of results*

Selection of participants

You must determine if the subject selection procedure and types of subjects used compromise the adequacy of the research Guidelines for your evaluation: - Was there sufficient detail provided regarding the participants to allow the reader to replicate the study? - Were inclusion and exclusion criteria reasonably justified?

Participants

You should evaluate: - HOW the participants were *sampled* and *selected* - Evidence of *sampling* or *selection bias* Any evidence of bias calls into question the validity of the results