The Methods Section of a Research Article

Cohen's effect size

Small effect = 0.2 Medium Effect = 0.5 Large Effect = 0.8

Category of reliability: Stability

Test-retest method - performing a complete repetition of the exact measurement and correlating results of the two measurements

Population

any group of individuals in which a researcher is ultimately interested

Reliability of Measurement

consistency or precision or accuracy of measurement

Two types of criterion validity

convergent divergent

Materials Section

describes what tools have been used to measure or generate the variables under study and identifies methodological threats to internal validity

For most ____ and ______ studies, it is not possible to investigate the entire population.

descriptive ; experimental

Informed Consent

participant voluntarily consents to participate in the research study after being made aware of the nature and purpose of the study

The American Psychological Association recommended subjects be replaced with _______ to imply that these individuals have consented to the study.

participants

Rosenthal Effect

problem of biased human observations

Confidentiality

the ability of other people to tie specific information or data to a given individual ; protecting the identity of a participant

Validity of a measurement

the degree to which it measures what it purports to measure

Construct Validity

the degree to which the measure reflects some theoretical construct or explanation of the behavior or characteristic being measured

Procedure

the researcher describes what is done to the subjects with the material ; "blueprint" describing the steps taken to develop, administer, and evaluate the research study

Research Protocol

the sequence of tasks performed by the subjects, manipulations of the independent variable, and the subsequent measurement of changes in the dependent variable

Data Analysis

typically, the last portion of the method section that describes how the data will be organized, summarized, and assessed

Observer Bias

when measuring or rating samples of behavior, judgement might be confounded

Divergent Validity

whether a test is independent of factors that are beyond the scope of the measured construct

Convergent Validity

whether a test yields a similar result to other assessments measuring the same construct

Ordinal Level of Measurement

Considers the identity and the magnitude of the attributes of objects or events by allowing us to rank these magnitudes from least to most E.G. class rank, customer satisfaction ranks

Category of reliability: Equivalence

Alternate or Parallel Forms - used to avoid the potential carryover effects associated with the test-retest method

Five factors associated with human observation that may influence the behavior of the research subjects:

Biosocial attributes of experimenters (age, sex, race, bodily activity) Psychosocial attributes of experimenters (personality characteristics) Situational variables (experimenter's experience, familiarity of subjects) Modeling effects (subjects behave as the experimenter does) Self-fulfilling prophecies (the experimenter's expectations and consequent treatment of subjects may influence the subjects' behavior)

Sources of measurement error

Characteristics of the examinee Behavior of the examiner-scorer Aspects of the test content Time factors Situation factors

______ are particularly important because statistical tests are so heavily influenced by sample sizes

Effect sizes

Content validity

How well does the content of the test items sample the behavior or characteristic to be measured? Subjective procedure for logically or rationally evaluating the measurements to see how well they reflect what the researcher wishes to measure

Ratio Level of Measurement

Include identity, magnitude, and equality of intervals and allow for specification of ratios between numbers

Interval Level of Measurement

Includes identity and magnitude and allows us to specify the equality of the intervals between adjacent examples of the attribute measured E.G. Temperature measurement with Celsius or Fahrenheit scales

Researchers can check an experimenter's reliability in 2 ways

Interexperimenter Reliability - the consistency among two or more experimenters in making measurement Intraexperimenter Reliability - the consistency of one experimenter in remaking a particular measurement

Why is the methods section important?

It's the structural framework or "blueprint" of a research article

Face validity

Not to what the test actually measures, but to what it appears superficially to measure E.G. does a test of reading comprehension "look" like a test of reading comprehension

Things that can affect research procedures

Poor test environment Unclear Instructions Observer Bias Data Analysis

Three basic principles for protecting human subjects and participants in research studies

Respect for persons:The ethical recognition of individuals as "autonomous agents" whose decisions are to be honored; Beneficence:The ethical obligation to protect individuals from harm and to "maximize possible benefits and minimize possible harms"; and Justice:The ethical requirement that the selection of individuals, as well as the distribution of "benefits" and "burdens," be fair and unbiased.

Nominal Level of Measurement

Simplest level of measurement where attributes of objects or events are classified into mutually exclusive categories E.G. Gender, handedness, favorite color, religion

Category of reliability: Internal Consistency

Split-Half - variation on alternate form reliability in which the two halves of measure may be seen as constituting two alternate forms. Requires that the items that constitute a given measure be split in half' each half is then correlated with the other measurement of the reliability coefficient Cronbach's Alpha & Kuder-Richardson 20 - provide reliability coefficients that estimate the average of all possible split-half correlations among the items of measure

3 categories of reliability

Stability Equivalence Internal Consistency

The researcher/clinician uses instrumentation to:

Standardize data-acquisition procedures Help acquire data under known conditions To provide permanent record of the data

Measurement Accuracy

Stems from the mathematical true score model True score is what the score would be under the ideal or perfect condition of measurements. However "ideal conditions" are unobtainable, thus there will always be some degree of error in measurement

Three main components of the methods section

Subjects (Participants) Materials Procedure

3 classes of reasons for poor measurement reliability

The person who is being measured may actually change from day to day The task may be different in two forms of the same measure of in different parts of one measure The measure may provide a limited sample of behavior that may not yield dependable characterizations of the behavior over the long run

Two basic evaluation questions need to be answered about materials

Was there adequate selection and measurement of the independent (classification, predictor) variable? Was there adequate selection and measurement of the dependent (criterion, predicted) variable?

The methods section of a research article describes . . .

Who or what was studied The materials that were employed How those materials were used to obtain data (Procedure) The research strategy/design used (how threats to validity are handled)

Privacy

an individual's ability to control when and under what conditions others will have access to personal information

Subject

a person upon whom an experiment is made

Sample

a subset of the population of interest Experimenters typically hope to generalize their results to the whole group, inferring that the data collected from the sample are similar to what they would have obtained from the entire population. If one wishes to generalize data to the majority of the population, then a large number of subjects will have to be used in the sample.

Belmont Report (1978)

document that serves as the foundation for the conduct of research in the United States

Institutional Review Board (IRB)

group established by college, universities, hospitals, and other institutions that conduct research to ensure that respect, beneficence, and justice are maintained for all human subjects

Measurement precision

measurement remains reasonably stable if the measurement procedure is repeated with the same subject

In communication disorders, at least a _____ effect size is typically required in order to detect clinically significant group differences or treatment gains

medium

Non-interactional effects

occurs when the observer does not actually affect the subject's performance but does affect the recording of the subjects behaviors

Interactional effects

occurs when the observer's interaction with the subject actually changes the subject's behavior during the experiment

When a choice of levels is available, the preferred order is . . .

ratio, interval, ordinal, and nominal

Systematic error of measurement

recur consistently with every repeated measurement

Criterion Validity

relates tests and measurements with each other