hca 465 chapter 5

coefficient alpha

aka "cronbach's alpha"; a measurement of internal consistency among a group of items (e.g., survey, test, or interview questions) -allows researchers to determine how well the items measure different aspects of the same topic -values range from 0 to 1; values closer to one indicate a higher internal consistency than values closer to zero

inter-rate reliability

aka "inter-observational reliability; the scoring or observations remain consistent regardless of the person who is doing it -investigators use a checklist to standardize and increase reliability by ensuring evaluators are collecting the same type of observational data

internal validity

focuses on the rigor of the study; its primary types are face validity, criterion validity, construct validity, content validitiy

measurement bias

happens during data collection by researchers or subjects because of systematic errors in measurement

relationship between internal and external validity

-INTERNAL validity is more critical than external -without internal validity, research is not testing what it reports to measure (however, as the study inclusion criteria becomes more selective, the results become less generalizable)

pilot test

-essential for both quantitative and qualitative studies -it is a complete dress rehearsal before data collection; involve every aspect including environment, data collection, content and outcomes of a study. -involves conducting a preliminary test of data collection tools and procedures to identify and eliminate problems -when executed correctly, they save time and resources during the actual data collection process

ways to reduce measurement errors

-pilot test -data collection training -double data entry -statistical consultation -triangulate data collection

how to conduct a pilot study by categories

-sample of respondents: recruit a small number (less than 10) of individuals with characteristics similar to the actual sample to test the method of recruitment. If it is difficult to recruit for the pilot test, than recruitment methods for the actual study need to be revised. -data collection: helps researchers determine if revisions are needed in the actual instrument or in the data collection procedures. Steps: 1) create the exact environment that will be used for the actual data collection 2) when possible, observe participants while they complete surveys during the pilot test. 3) after the surveys or interviews are complete, ask the participant a few questions -data analysis: enter the data and conduct a pilot test of the data analysis procedures -outcome: conduct the actual study

benefits of pilot testing

-taking the time to conduct a thorough pilot test is always worth the time and money, because once data is collected, it is too late to fix mistakes that could prove to be fatal flaws in the entire study -when reporting results, investigators document the pilot study and revisions made based on results, adding authenticity

randomized controlled trial designs

-the gold standard of research design -participants are randomly assigned to either a treatment of placebo group -participants have similar characteristics (age, gender, or length of diagnoses) -allows researchers to draw conclusions with confidence if one group is significantly different at the end of the study due to random assignment -controlled study (one treatment and one placebo group)

what 3 areas should one investigate when looking for systematical errors?

1) environment 2) observation 3) drift *the problem with each type of error is even when the error is recognized, it is difficult to determine when it began in the data collection process and how much the systematic error influenced actual results

factors that influence generalizability (external validity)

1) population (also known as selection and treatment): when population selection is too specific, treatment is matched to a specific sample and is not applicable to a wider population 2) environment: studies conducted in particular spaces/settings 3) temporal/sequential factors: the time of year/season a study was conducted in can affect the results 4) participants: animal-to-human links; human-to-human links; gender bias; racial bias; cultural and ethnocentric bias 5) testing and treatment interaction: if participants learn from the pretest, they may be less likely to learn as much from the treatment 6) reactive arrangements (hawthorne effect): if individuals change their behavior when observed (threat to internal validity), results are not generalizable to real-world conditions (threat to external validity) 7) multiple treatment conditions: the same individuals exposed to multiple treatments; because multiple treatments may create an artificial setting that does not exist in the real world, results may not be generalizable *additionally, researchers and evaluators must be aware of all possible threats that affect INDEPENDENT variables

Validity

the extent to which a test measures what it purports to measure

split-half technique

a technique to measure homogeneity by dividing the entire test or survey into two equal halves (e.g. odd numbered and even-numbered questions). -The two forms are administered to the same individuals. -if the odd-numbered questions yield the same results as the even-numbered, the entire test is deemed reliable

Reactive Arrangements (Hawthorne Effect)

applies to external validity as well; if the individuals change their behavior when observed (threat to internal validity), results are not generalizable to real-world conditions (threats to external validity)

what is reliable research based on?

based on several basic assumptions, including randomized controlled trial designs and adequate sample sizes, and is free of known bias

observation

can change due to observer fatigue, time of day, room temperature, training of different observers, attitude of participants, and various other human behavior variables

Test-retest technique

checking the stability of an instrument by giving it to an individual and then giving it to her again after a certain period of time -involves two assumptions: 1) the item or observation does not change over time 2) the time between the first and second administration is long enough

reliability

consistency of the instrument or survey being used, not the respondents -related to consistency or the ability to repeat results -important because if a test's results are different each time, the test is faulty and can lead to inaccurate conclusions -can be established one of two ways: stability and internal consistency

external validity

deals with generalizability; if evaluation or research is repeated with different populations, situations, time, or environments, the results are expected to be the same -a threat to this might explain how generalizations are incorrect -repeating evaluation methods and research in different populations is the best way to access generalizability

controlled study

defined as one group receiving treatment while another, similar group receives no treatment -may be randomized from original groups of participants, or may be similar groups at a different location

systematic errors

errors that are consistent in the same direction, so that no matter how many times the experiment is repeated, the same errors occur -validity is influenced by these kind of errors -these errors introduce inaccuracy to the measurement, cause bias in the data, and diminish the extent to which the test is measuring what it purports to measure -problematic to detect and eliminate -it is not possible to reduce the effect of these kind of errors through statistical methods

double data entry

evaluators and researchers enter a portion of the data twice and then compare the two entries. -if there are multiple errors, investigators go back to the original data and determine why the errors are occuring

relationship with similar measures

ex: researchers are interested in developing a shorter depression scale for adolescents. Adolescents are asked to complete both the older, longer depression scale with established validity and the newly developed, shorter depression scale. If the measurement of the established depression scale are similar to the measurement of the new scale, then researchers conclude evidence of construct validity.

face validity

exames how the test appears *NOT based on theory but merely the appearance of the test (potential ease of completion, comprehension, and readability) -can be compared to "showing up in the correct outfit" or "looking the part"; if the survey does not look appealing, individuals are less likely to complete it -also refers to the logical sense of the survey (do the questions relate to the subject being addressed?)

content validity

how well a test measures the specific content it is intended to measure

testing and treatment interaction

if participants learn from the pretest, they may be less likely to learn as much from the treatment -if a participant is sure that they received a perfect score on the pretest, they are less likely to listen intently during the class, because they already know the information

relationship between reliability and validity

if the scale is VALID, the results should be the same if the same person tested on it over and over. It measures what it is intended to measure HOWEVER, the reverse is not always true reliability is about consistency and repeatability

types of validity

internal validity (face validity, criterion-related validity; construct validity, content validity) external validity (

statistical consultation

investigators consult with statisticians to seek assistance with entering data and determine ways to reduce and/or measure data errors

intervention bias

involves how many intervention groups are treated differently than control groups if the researchers involved know which group is which

Is it more important for a test to be reliable or valid?

it is more important for a test to be valid than reliable

criterion-related validity

measures one topic in two different ways *most common in having a written test and then a test that applies the skill (like a written test for your permit and a behind-the-wheel driving test for your license) *it is important that the learner knows the didactic information as well as how to perform the skill

inter-rate reliability score

number of concurrences / number of opportunities for concurrence x 100 ex: investigators use a predetermined checklist to observe 80 patients in the clinic waiting room; according to their checklist, they agree on 68 out of the 80 ratings. The calculation is 68/80 = .85 x 100 = 85% -therefore, the investigators agree 85% of the time, or their observations have a reliability of 85%

random errors

occur by chance and are inconsistent across the respondents -these errors increase or decrease results in an unpredictable manner; therefore, researchers and evaluators have no control over the occurrence of them they influence consistency in several ways: 1) participating INDIVIDUALS may change from day 1 to day 2. 2) TASKS may change between day 1 and day 2 3) if there is a small sample of participating individuals, outside forces have a greater effect on the outcome *each alters the reliability of knowing whether the experiment was reliable in doing what it intended to 4) these errors may also occur in written tests. There are 3 examples under this source: 1) if the test is TOO SHORT, individual scores are based more on chance and luck than knowledge 2) if the test is NOT GRADE precisely the SAME WAY for each student, these errors can cause inconsistent reliability 3) if the test is NOT ADMINISTERED CORRECTLY Ways to reduce these kinds of errors: 1) have a larger sample size; these kind of errors are less influential in either direction when the sample size increases 2) average scores over a larger sample size to reduce these errors through statistical methods

selection bias

occurs if specific individuals or groups are purposely omitted from the investigation

bias

occurs in research and evaluations from numerous causes including selection, measurement, and intervention.

relationship with experimental measures

patients are asked to complete two anxiety scales: one with established validity and a newly developed scale. In the midst of potential anxiety, patients should score higher on both scales. If both scales show similar levels of anxiety, researchers should conclude evidence of construct validity

environment

refers to how the setting changes over the course of research -can be a source of systematical errors ex: if the research was conducted outside, the temperature, humidity, wind speed, or heat index may have caused variations in data and results that are difficult to pinpoint

cultural and ethnocentric bias

research conducted on one specific cultural group is not generalizable to other cultures. for example, spanish speaking individuals do not share the same cultural background (i.e. there are differences among the cultures in Puerto Rico, Mexico, Nicaragua, Honduras, Venezuela, and Spain) -religions can also contribute to cultural differences among various populations

racial bias

research conducted only on African Americans, for example, does not yield generalizable results to other racial groups. In today's world of linking genetics to health conditions, issues of racial inclusion are more important than ever.

gender bias

research including only men or only women is not generalizable to the non-represented gender group. The same is true for studies that include only heterosexual individuals and fail to recognize LGBTQ individuals.

comparison scores among defined groups

researchers create a new dexterity aptitude test with the hypothesis that adults have varying degrees of dexterity. *more than 1000 adults complete the new test. If results show that adults in specific fields (e.g. surgeons, dentists, musicians) score higher on the new dexterity aptitude test than adults in non-dexterity applications, there is evidence of construct validity

human-to-human links

researchers use college students for many studies because the students are a convenient sample. However, results are questioned because data gathered from college students may or may not be generalizable to other groups of young adults not attending college.

internal consistency

the extent to which each question in a survey is related to the same topic; also called "homogeneity" -in quantitative research, researchers use a split-half technique to measure homogeneity -Test A and Test B scores should be approximately equal, with all questions measuring concepts related to the material covered in the tested chapter

randomized

the research participants are randomly assigned to either a treatment group or a non-treatment group (placebo group) -allows researchers to draw conclusions with confidence if one group is significantly different at the end of the study

multiple treatment conditions

the same individuals exposed to multiple treatments *because multiple treatments may create an artificial setting that may exist in the real world, results may not be generalizable

construct validity

used to measure a concept that is not actually observable *can be established by exploring relationships with similar measures, experimental measures, and comparison scores among defined groups

threats to internal validity

things that confuse or confound test and survey results and overall findings 9 key threats: 1) history: an event happens during research that influences the behavior of participating individuals 2) maturation: the natural changes that occur over time with individuals 3) testing: differences noted from pretest to posttest that can be attributed to students becoming familiar with the test 4) instrumentation: measures changes in respondent performance that cannot be credited to the treatment or intervention 5) regression: some respondents performing well on pretests and poorly on post-tests or vice versa, merely by change. sometimes manifests as "regression to the mean" in which data from these high-pretest to low-posttest performances cancel each other out, and the overall score is similar to the average 6) ceiling effect and floor effect: **ceiling effect: when all participating individuals perform extremely well on pretest and posttest, therefore making it difficult to determine any changes the intervention may have had **floor effect: when individual performance starts out low and remains low; leads investigators to think that individuals are unresponsive to treatment, when in fact the low performance may be caused by a factor outside the intervention or treatment altogether 7) attrition: individuals lost from the study; if a large number of individuals leave the study for a variety of reasons, results may reflect more about the individuals who stayed in the study than the treatment conditions of the study 8) selection: when participating individuals are different at the onset of the study; it can make it difficult to know whether the type of condition or the treatment used influenced the results 9) hawthorne effect: improving performance when you are aware that you are being watched

triangulate data collection

to increase reliability and validity, researchers and evaluators choose to collect similar data using more than one method

determining adequate sample size

too small: results are inconclusive and significant differences among groups are statistically harder to determine too large: cost, feasibility, and time become problematic *an ideal sample size to adequately represent the target population is what researchers strive for. However, a larger population is preferred over a smaller one because of the increased precision and accuracy of the study

data collection training

training data collectors for consistency (e.g., detailed checklists, machine measurement calibration checks, inter-rater comparisons to avoid creating errors)

measurement errors

two types of errors that influence the results of surveys, tests, and instruments: random errors and systematic erros

dependent variables

variables that are fixed and not manipulated

independent variables

variables that researchers manipulate and control

advantages and disadv of estimating reliability

ways to establish reliability: stability --> test-retest *pros: single rater is adequate, no need to train teams of raters, less expensive and time consuming *cons: often difficult to recruit same respondents to respond twice; individuals may not respond as seriously the second time ways to establish reliability: internal consistency --> quantative: split half forms *pros: respondents take both surveys at the same time; no need to recruit respondents twice *cons: need to create a large pool of items qualitative: inter-rater or inter-observation reliability *pros: best for observational research, especially when video recording is used; possible option: one rater reviewing video at two different times *cons: expensive, time consuming, requires a team of raters for best results

drift

when evidence suggests that the dat is slowly moving in one direction -ex: may occur if the machine used for lab results is not calibrated each day prior to running complete blood count samples

animal to human links

when researchers use rats or other animals to test specific drugs, it is questionable whether humans will react to the new drug in the same way as rats

stability

when the results of a survey or instrument are consistent over time.