Part 3 Test 3 Psychology 370

Good surveys share the following characteristics:

-specific and measurable objectives -straightforward questions that can be understood by most -pretested to ensure clarity and avoidance of incorrect skip patterns -administered to adequate sample of respondents so generalizations can be made -appropriate analysis to obtain the objectives -accurate reporting of results (both verbal and written) -reliable and valid

What are the three types of survey types discussed?

1. Self administered 2. Personal Interviews 3. Structured record reviews

Defining the Audience is part of which step of the test development? a. 1st b. 2nd c. 5th d. 3rd e. 4th

1st

Constructing the Survey (2nd phase) involves ___________.

Choosing the type of survey.

What is the name of a well known household survey?

Current Population Survey

Cognitive strategy up ahead. What does CASTRO strand for?

Dropping the T and we get CASRO. It stands for Council American Survey Research Organization.

What are two methods used for collecting scientific information?

Experimental research and descriptive/ correlational research

In bold, one of the purpose of pretesting is to:

Identify sources of non sampling measurement errors.

What are 6 types of scales?

Qualitative Extent Frequency Developmental Comparison Performace

Which of the following steps is False regarding pilot test? a. Setting up the pilot test b. Conducting the pilot study c. Analyzing the results d. Communicate results

d. communicate. The three steps of the pilot test include: Setup, conduct and analyze.

Closed-ended questions provide primarily_________, and are frequently used in _____________.

quantitative data, confirmatory

Purpose of pretesting is to

1. Identify sources of non-sampling measurement errors—errors associated with the design and administration of the survey 2. Examine how effective revisions of a question or an entire survey are Indicate the effect of alternative versions of a question or survey 3. Assess the final version of a questionnaire for respondent understanding, time to complete, and ease of completion 4. Allow the survey analysts to obtain data from the survey and to make changes to the format that might make data entry or analysis more efficient

What steps are in the 1st phase of preparing a survey?

1.Identifying the objectives What is the purpose of the survey and what will it measure? 2. Defining the objectives operationally Operational definitions are specific behaviors that represent the purpose 3. Constructing a plan Purpose and what survey will measure

Defining the Test purpose is part of which test of the test development? a. 1st b. 2nd c. 5th d. 3rd e. 4th

1st

Defining the Testing Universe is part of which step of the test development? a. 1st b. 2nd c. 5th d. 3rd e. 4th

1st

Which step Provides foundation for all other development activities?

1st

Selecting appropriate respondents, choosing sample size and distributing the survey are part of phase _____.

3, Administering the survey

The pilot test is a a. descriptive way of testing reliability and validity b. are very simple and easy to conduct c. scientific investigation of test reliability and validity for its purpose d. all of the above

A pilot test is a scientific analysis of test's reliability and validity based on it's purpose.

A valid percent of .64 means that_________ a. 64% people got the question wrong b. 36% people got the question right c. 64% people got the question right d. 36% people thought the question was too difficult

A valid percent of .64% in the "correct" row. Means that 64% of people got the question right.

A valid survey yields _________.

Accurate information and various methods for determining validity.

Defining the test universe Involves: a. preparing a definition of the construct to be measured c. Review of literature to identify how construct can be defined d. Review of other tests that measure similar construct e. all of the above.

All of the above

Good survey questions ___________. a. use appropriate rating scale and response options. b. include appropriate categorical alternatives c. Use correct syntax d. purposeful and straightforward e. all of the above.

All of the above.

Which of the following statements is false about setting up the Pilot Test? a. Confidentiality rules must be observed and and results are reported in summary form (aggregate data). b. Sample must be representative of target sample c. Pilot test setting must be similar to actual work setting d. Participants dont' need to know they are participating in a research study.

Answer: d. Correct answer: Participants need to know they are part of a research study. Test setting must mimic actual work setting. Researcher must obey confidential rules Results to be reported in aggregate data (summary of results) Pilot study participants must be representative of target group

We acquire information from a highly respected source. For example, we come to believe that one brand of aspirin is better than another brand because a pharmacist tells us so.

Authority

What kind of question requires participants to choose from a limited number of responses predetermined by the researcher?

Closed ended questions

The 4th phase of the survey research method process includes:

Coding the survey questions Entering data into spreadsheet program Verifying data are entered correctly Conducting statistical analysis Interpreting the results

What do survey research firms do?

Company that constructs (creates), administers (gives), and analyses surveys.

What is the Cognitive Aspect of Answering Questions?

Comprehension Retrival Judgement Response/communication

Using whatever group is most available is an example of _______. a. simple random sampling b. systematic sampling c. stratified sampling d. cluster sampling e. convenience sampling

Convenience sampling

The Code of Standards and Ethics for Survey Research is published by _________________.

Council American Survey Research Organization

What is descriptive or correlation research used for?

Describe/compare situations/phenomenon Most surveys are descriptive

We acquire information through personal experience. For example, we come to believe that one brand of aspirin reduces headaches more effectively because we have used it and it works for us.

Empirical

True or false. Analyzing test results helps gather information on qualitative but not quantitative data.

False. Analyzing test results helps gather data on both qualitative and quantitative data.

True or false. Breadth and depth of a pilot study is the same for all pilot studies.

False. The breadth and depth of the pilot study varies depending on the size and complexity of the target sample.

True or false. Ambiguous questions are make better survey questions.

False. Good survey questions are unambiguous.

True of false. good survey questions (1 itme) address multiple questions.

False. Good survey questions ask one and only 1 question.

True or False. Test are only developed in the United States.

False. Tests are developed and used worldwide

What is the comfortable reading level for good survey questions?

Good survey questions are typically at a 6th grade reading level

Sampling error refer to the amount of ______ that can be attributed to the ____________ representation of the ______________ by the sample of respondents chose.

How much error can be attributed to lack of representation of the target population by the sample of respondents chosen

In the test item table from the item difficulty analysis, "valid" p = ___________. a. number who responded incorrectly/ total respondents b. number who responded correctly/total respondents c. number who responded incorrectly/degrees of freedom d. number who responded incorrectly/ degrees of freedom e. percent that the result was due to chance

In the test item table, p is the number of people that got the answer right, divided by the number of people.

We acquire information without any reasoning or inferring. For example, we come to believe that one brand of aspirin is more effective than another brand just because we believe it is so.

Intuition

According to Helmstadter, what are the 6 ways we acquire knowledge? *Cognitive strategy: TIARES, like TIARAS but with an E.

Knowledge Acquisition We obtain knowledge in at least six different ways: Tenacity Intuition Authority Rationalism Empiricism Scientific method

Errors associated with how a survey performs in a particular poupulation

Measurement erros

One-on-one interviews Respondent focus groups Behavior coding respondent-interviewer interactions Interviewer and respondent debriefings Split-sample tests Item nonresponse rate are all _________________.

Methods for pretesting

What is your current marital status? (Select one.) Single Married Divorced Separated Widowed

Multiple choice question

What are the 5 basic types of closed ended questions:

Multiple-choice Categorical Likert-scale Numerical Ordinal.

True or false. A population is a subset of a sample.

No, a sample is a subset of a population.

True of False. Constructing reliable and valid surveys is a quick process.

No. Constructing reliable and valid surveys takes time

Convinence sampling is an example of ________. a. Probability sampling b. simple random sampling c. stratified sampling d. non probability sampling

Non probability sampling

Non sampling measurement errors refer to the errors associated with: a. errors in the sample b. errors with the design and administration c. errors due to the lack of randomization in testing d. errors due to the researcher and participant

Non sampling measurement errors refers to the errors associated with the design and administration.

True or false. "Check all that apply scale" present no challenges to researchers.

Not true.

What is the order of the survey report outline using items listed below:

Objectives Methods Findings Implications

What kind of question allows participants to respond in any way they choose?

Open ended questions' Eg: What is your current marital status? A: [Participants provide answers in their own words]

Types of research questions:

Open-ended questions Closed-ended questions Yes/No questions Fill-in-the-blank questions Implied no choice Single-item choices Enfolded formats Free choices Multiple-choice format Ranking questions Rating questions Guttman format Likert and other intensity scale formats Semantic differential format Paired comparisons and constant referent comparisons

Face to face and telephone discussions surveys between researcher and participant are a type of __________. a. structured record reviews b. personal interviews c. individual surveys d. self-administered

Personal interviews

A pilot test helps gather additional data such as: a. criterion measure b. length of time to complete test c. who your neighbors really are d. a and b

Pilot test help gather data on criterion and test time.

Regardless of purpose of survey, it is important to carefully

Prepare for the survey Construct the survey Administer the survey Enter and analyze survey data Present the findings

What is the 5th phase of the survey research method?

Present the findings Reporting results to those who commissioned the survey's development and sometimes to the public at large

What does the Council American Survey Research Organization do?

Provides education of ethical standards for research and surveys.

We acquire information through reasoning. For example, we come to believe that one brand of aspirin reduces headaches more effectively than another brand because the first brand includes ingredients that are similar to another medicine we use that really works for us.

Rationalism

Why do researchers try to get a p value that averages to .5 in difficulty? a. it's the most difficult level and researchers want to weed out dumb students b. it' an easy difficult level that allows all students to pass c. gives least variability in test scores d. give the most variability in test scores

Researchers like variability, and the .5 difficulty level has the most variability.

Researchers try to get a p values that averages to ______ in difficulty level. a. .9 to 1 b. .5 c. 0 to .02 d. .3

Researchers try to get a p value that averages to .5 in difficulty.

A process for generating a body of knowledge that allows us to obtain more accurate information through systematic and objective observations

Scientific Method

We acquire information by testing ideas and beliefs according to a specific testing procedure that can be observed objectively. This method is without personal beliefs, perceptions, biases, values, attitudes, and emotions. For example, we come to believe that one brand of aspirin is more effective at reducing headaches because systematic research has accumulated evidence of its effectiveness.

Scientific Method

Mail and individually administered surveys are a type of __________. a. structured record reviews b. personal interviews c. individual surveys d. self-administered

Self-administered

What are the steps associated with the scientific method?

Steps associated with the scientific method: Identify problem and form hypothesis Design study Conduct the study Analyze and interpret data Communicate research results

Researcher divides the entire population into different subgroups, then randomly selects the final subjects from the different subgroup. Every population member has an equal chance of being in the sample. a. simple random sampling b. systematic sampling c. stratified sampling d. cluster sampling e. convenience sampling

Stratified sampling

What are surveys?

Surveys are tools used to collect information about a attitudes, behaviors and knowledge.

Surveys are thought to be a science and art. Why?

Surveys follow a scientific process. It's also an art form because there is effort put into the it's design and writing good questions takes years of practice.

We acquire information based on superstition or habit, leading us to continue believing something we have always believed. For example, we come to believe that walking under a ladder will bring us bad luck

Tenacity

Testing universe is: a. body of knowledge or behaviors of respondents b. testing the universe that we live in c. body of knowledge or behaviors represented by the test d. none of the above

Test universe if knowledge or behaviors represented by the test .

What is the goal or business strategy of the Council American Survey Research (CASRO)?

The goals of the CASRO are to: Protect Represet Communicate Educate

What are the step for Designing Surveys?

The steps in designing a survey are: Presurvey issues construct survey Administer survey Analyze data Communicate findings

What is one disadvantage about surveys?

They don't normally help us determine cause and effect.

To catch respondents' attention and motivate respondents to complete survey so as to reduce respond errors, pay attention to...

Title and seal Appeal and instructions Headings and subheadings Transitions Response directions Bold typeface Justification of response spaces Shading White space Printing Font type *In a nutshell the presentation of the survey.

True or false. Plan analysis at time of survey construction

True

Operational Definitions for a Class Evaluation Survey Example. Ojective is to measure the instructor's knowledge of subject matter

Was the instructor knowledgeable about the material? Did the instructor show enthusiasm for the material?

Coding, Entering, and Analyzing Survey Data is part of which phase of the survey research method?

a. 1st b. 2nd c. 3rd d. 4th

Every population member has an equal chance of being selected. Although this method does not guarantee that the sample will be representative. a. simple random sampling b. systematic sampling c. stratified sampling d. cluster sampling e. convenience sampling

a. simple random sampling

Item Difficulty refers to the a. the difficulty level of the test question b. percentage of test takes who reponde incorrectly c. percentage of test takers who respond correctly d. the ceiling or floor effect as a result of item difficulty

c. Item difficulty is the % of students who answered a question correctly.

Experimental research help scientist determine _______.

cause and effect.

A population is divided into groups, then members are selected from those groups. a. simple random sampling b. systematic sampling c. stratified sampling d. cluster sampling e. convenience sampling

cluster sampling

A reliable survey yields___________.

consistent data

What are methods used for determining validity?

content validity criterion validity construct validity face validity

Multiple choice questions are used when you want your respondents to: a. Have freedom b. choose what you want them to choose to improve your survey results c. Help participants with predetermined ideas d. choose the best possible asnwer

d. choose the best possible aswer

Most surveys are _________. a. experimental b. nonsense c. descriptive d. correlational

descriptive

sample size _________ as the confidence level increases from 95% to 99%.

increases

Probability sampling_________. a. non everyone has an equal chance of being selected. b. is a method for selecting a sample c. uses statistics to ensure sample is representative. e. b and c.

is a method of sample selection and uses statistics to make sure sample is representative.

Non-probability sampling ________. a. not everyone has an equal chance of being selected. b. is a method for selecting a sample c. uses statistics to ensure sample is representative. e. a and c

is a method of selecting a sample in which not everyone has an equal chance of being selected.

Unpredictable errors are_______

random erros

What are some important statistics when analyzing the data:

response rate reliability sampling error univariate analyses Frequency counts, means, modes, and medians bivariate and multivariate analyses E.g. correlation coefficients, cross-tabulations, chi-square, t-tests, analyses of variance

Selecting every 12th person to be part of the sample is an example of __________. a. simple random sampling b. systematic sampling c. stratified sampling d. cluster sampling e. convenience sampling

systematic sampling