REF 469 Test 2 Review

1. Clearly indicate cognitive skills or process that should be employed to formulate the answer, 2. Clarify the pupil's task with concise details 3. Set appropriate page or time limits.

*A WELL-CONSTRUCTED ESSAY ITEM should...

•DEBATES •MOCK TRIALS •GIVING A CLASS A PRESENTATION •REENACTMENTS •SCIENCE EXPERIMENTS •MUSICAL PERFORMANCES

*Examples of Performance Assessments:

performance-based

*Teachers establish situations that allow them to observe and rate learners directly as they analyze, problem solve, experiment, make decisions, measure, cooperate with others, present orally, or produce a product in what type of test

they use DIRECT measures of learning rather than indicators

*The MAJOR ADVANTAGE of performance tests is that

Taxonomy

*The __________ of educational objectives for the cognitive domain helps categorize objectives at different levels of complexity.

•to test a student's ability to use higher level cognitive skills/mental processes •When objective test items are inappropriate •Few test items are necessary •Test security is in question

*What are ESSAY items best used for?

Instructional objective

*____________ should be stated in observables, behavioral terms (using an action verb), in order for two or more individuals to agree that a student has or has not displayed the learning outcome in question.

Multiple choice items

*can measure behavior at higher levels of taxonomy (ex: comprehension, synthesis, evaluation) but are most difficult of the objective items to construct and should be used with caution on younger children

The affective taxonomy

*describes objectives that reflect underlying emotions, feelings, or values, rather than cognitive or thought complexity.

CHECKLISTS

2 VARIATIONS/characteristics in student behavior are being observed - it's either PRESENT or ABSENT. They are best suited for complex behaviors or performances that can be divided into a series of clearly defined, specific actions.

•Can assess complex learning objectives •EASY to construct •Can assess communication skills •They eliminate student guessing

ADVANTAGES of essay items

•What is the goal of your test? Link directly to objectives •Know the odds of students passing by guessing •*Objective Test items should also be objective in scoring

Advantages and disadvantages of the different formats are (these should influence choice):

Conveys to the teacher the... •# of items to be constructed per objective •level of taxonomy for each item •*Whether the test represents a balanced picture of the objectives that were taught

Advantages of using a test blueprint

Students have to develop positive dispositions and "habits of mind" which include behaviors such as: •Constructive criticism •Tolerance of ambiguity •Respect for reason •Appreciation for the significance of the past

Affective domain and social domain

SPECIAL CONDITIONS

An instructional objective describes any ___________ in which the learning will take place.

1. Be STUDENT DIRECTED (students will be able to...) 2. Contain an OBSERVABLE BEHAVIOR or an ACTION VERB specifying the learning outcome 3. State any SPECIAL CONDITIONS under which the behavior must be displayed 4. State a PERFORMANCE LEVEL considered to be indicative of mastery - it must be MEASURABLE

An instructional objective should (four steps):

1. What kinds of essential tasks, achievements, or other valued competencies am I missing with paper-and-pencil tests? 2. What accomplishments of those who practice my discipline are valued but left unmeasured by conventional tests? •Skills related to acquiring information •Skills related to organizing and using information

Cognitive domain - ask the following questions to guide selection of objectives

•Absolutes in wording (i.e. always, all, never, only) •Double negatives•Requiring and opinion •Double-barreled statements (i.e., use of and, or) •Excessive wordiness •A tendency to reflect statements taken verbatim from readings (w/o context, or out of context)

Common pitfalls of true-false items

Percentages •# of items correct •# of consecutive items correct or consecutive errorless performances •Essential features included •Completion within a prescribed time limit •Completion with a certain degree of accuracy

Criterion levels may be stated as

•Takes longer to score/grade •Scoring can be unreliable-essay tests have lowest "INTER-SCORER RELIABILITY" •Essays sample limited content •Answers are susceptible to bluffing

DISADVANTAGES of essay items

Practicing the violin; Practicing multiplication tables; Watching the evening news

EXAMPLES of learning activities

•*Objectives specify higher-level cognitive processes •When objective test items such as T/F or Multiple Choice are inappropriate •Few tests or few test items are necessary •*Test security is a question

Essay items should be used when...

In the field of athletics, diving and gymnastics are examples of performances that judges rate DIRECTLY; at band contests, judges DIRECTLY see, hear, and rate students as they perform

Example of a performance-based test

GIVEN A CALCULATOR, students will be able to multiply two-digit numbers, correct to the nearest whole number

Example of stating conditions

Be familiar with the law; Appreciate art deco; Enjoy speaking French

Examples of UNOBSERVABLE LEARNING OUTCOMES

Given 20 two-digit addition problems, the student will be able to compute at least 18 OUT OF 20 answers correctly. Given 20 two-digit addition problems, the student will be able to compute at least 90% OF THE PROBLEMS correctly.

Examples of criterion levels

Fixing a car radio; Adding signed numbers correctly; Punctuating an essay correctly

Examples of learning outcomes

Circle the initial sound of words; Recite the names of the characters in Tom Sawyer; Change the spark plugs on an engine

Examples of observable learning outcomes

1. Deciding WHAT to test 2. Designing the assessment context 3. Specifying the scoring rubrics

First 3 steps in constructing a performance assessment

focus, brevity, and clarity

For objective items, consider:

•*Use pictorial, graphical or tabular stimuli (Pictures, drawings, graphs, tables, maps and so on require the student to think at the application level of Taxonomy of Educational Objectives and may involve higher cognitive processes.; Use of these stimuli can generate several items at this level) •Use analogies that demonstrate relationships among terms •Require the application of previously learned principles or procedures to novel situations

Higher level multiple choice questions

Create a list of objectives that specifies the knowledge, skills, habits of mind, and indicators of the outcomes that will be the focus of your instruction.•

How to decide what to test (step 1)

Instructional objective

If the observable learning outcome is to take place at a particular time, in a particular place, with particular tools or materials, etc., this must be stated in the

instructional objectives, instructional activities, and test items.

In the 3-stage model of classroom measurement to enhance CONTENT VALIDITY we need alignment between

•Questions that use tabular data, charts, and graphs that come from the text as well as newspapers, magazines, and reports related to the curriculum •Require students to apply think about and apply information that comes from real-life situations

OPEN-BOOK questions and exams are ideal for...

cognitive, affective (attitudes, social skills), or psychomotor processes •assessments of processes, products, or both.

Performance tests can measure

•Too many blanks •Lack of specificity (too many potential responses) •Failure to state a problem

Pitfalls of completion items

•True-False items •Multiple Choice items •Completion items

Possible formats include

•*Grammatical cues or specific determiners 9ex. The use of "a" or "an") •Multiple defensible answers •Unordered option lists •Stem clues •Opinionated statements •Lack of a problem statement in the stem •Redundant wording •Wordiness in the correct option

Possible pitfalls of multiple choice include:

This is a technique used to help instructors remember to write objectives and test items at differentcognitive levels (of Bloom's Taxonomy) to ensure that tests include items that measure higher-level thinking.

Purpose of the test blueprint

brevity

Should be as succinct without sacrificing...

1.What does the doing of ___ look and feel like to professionals? 2.What projects and tasks are performed by these professionals? 3.What roles - or habits of mind - do professionals assume?

Some questions to ask when designing the assessment context are

A particular time; A particular place; Special equipment that is used, tools, or other resources

Special conditions may include

criterion levels

Stating the __________ is the part of the instructional objective that indicates HOW WELL the behavior is to be performed; MEASURABLE part of the objective

•ID cognitive processes you want student to use BEFORE writing the test item •*Clearly state the task including grading criteria in writing and orally •Avoid beginning essay items with what, who, when, and list unless you are only measuring at knowledge level •Ask for evidence/support for a controversial position •Avoid using optional items (have everyone answer all essay items-no picking and choosing) •Establish reasonable time and/or page limits •Use essay Q's for measuring learning outcomes that cannot be adequately measured by objective items (T/F, multiple choice, matching...) •Be sure item matches instructional objective

Suggestions for WRITING essay items

•Write good essay items •Using Restricted Range (time/page limits) whenever appropriate •Using an ADVANCED RUBRIC or PREDETERMINED SCORING SCHEME •Consistent scoring with ALL students •GRADE ANONYMOUSLY by removing or covering names to avoid bias •Scoring ALL student responses to ONE ITEM before moving to the next one •Keep scores from previous items hidden when scoring other items •Rescoring papers before returning them and averaging discrepant scores

Suggestions for more objective/reliable SCORING of essay items...

1.If possible, items should require a single-word answer/brief and definite statement. 2.Be sure the question/statement poses a problem to the examinee. •A direct question is often more desirable than an incomplete statement. 3.The answer required is factually correct. •Watch the language is precise and accurate in relation to the subject matter 4.Omit only key words •Don't eliminate so many elements that the sense of the content is impaired. 5.*Write the blank as near the end of the sentence rather than near the beginning. •This will prevent awkward sentences. 6.If the problem requires a numerical answer, indicate the units in which it is to be expressed

Suggestions for writing completion or supply items

1.The stem of the item should clearly formulate a problem. •Include only material to make the problem clear/specific •Include as much of the item as possible •keep the response items short 2.*Be sure that there is one and only one correct/clearly best answer 3.Be sure wrong answers choices are plausible. •Eliminate unintentional grammatical clues •Keep the length and form of all the answer choices equal •Rotate the position of the correct answer from item to item randomly 4.Use negative questions or statements only if the knowledge being tested requires it •It is more important for students to know what it is rather than what is not 5.Include from 3-5 options (2-4 distractors w/ the correct answer) •optimize testing for knowledge (not) encouraging guessing. •Not necessary to have the same number of answers for each item 6.*To increase the difficulty of a multiple-choice item, increase the similarity or content among the options 7.Use the option "none of the above" sparingly and only when the keyed answer can be unequivocally as right or wrong. 8.Avoid using "all of the above". It is usually the correct answer and makes the item too easy for students with partial information.

Suggestions for writing multiple choice items

1.Very basic - instructions as to how to mark T/F should be clearly explained prior to the test 2.*Make certain statements are definitely either true or false (not opinion) (focus) •If using opinion, attribute it to a specific source 3.Use relatively short statements and eliminate extra material (brevity) 4.Keep statements about the same length 5.Use approximately equal numbers of true and false items 6.Avoid using double-negative statements (clarity)

Suggestions for writing true-false items

Performance-based tests

Tests that are often used in science, social studies, or math

Clarity

The meaning of the item should be crystal clear to all test-takers. The goal is to reduce error in the measurement process.

•writing appropriate questions that elicit complex cognitive skills •structuring the student's response so that other factors do not obscure your ability to evaluate it. •(ex. Writing skills of student)

The potential of the essay question as an evaluation device depends on

1.What knowledge or content is essential for learner understanding of the subject matter? 2.What intellectual skills are necessary for the learner to use this knowledge or content? 3.What habits of mind are important for the learner to successfully perform with this knowledge or content?

Three general questions to ask when deciding what to teach

Restricted response item and extended response item

Two main types of essay items

OBSERVABLE

Two or more individuals observing a student should be able to agree that the student demonstrated the desired learning outcome, therefore, instructional objectives must be __________

•Hands on exercise •Problems to solve •Observable outcomes or product •A process that can be observed

What to provide when designing the assessment context

Learning outcomes

_________ are ends (products); implies a specific product or result

LEARNING ACTIVITES

__________ are the means (processes) to the ends; only implies an activity that will lead to a learning outcome

Instructional

_____________ objectives should include the level of proficiency to be demonstrated and the special conditions under which the skills must be demonstrated.

Performance test

______________ is being advocated as a fairer and more relevant alternative to conventional testing.

Instructional Objective

a clear and concise statement of the skill or skills (outcomes) that your students will be expected to perform after a unit of instruction.

RUBRICS

are CRITERIA FOR GRADING/SCORING performance tests. They are samples of acceptable responses against which the rater compares a student's performance.

Rating scales

assign numbers to categories representing different degrees of performance. They are typically used for those aspects of a complex performance such as: attitudes, products, and social skills, that do not lend themselves to yes/no or present/absent type judgments.

Holistic scoring

estimates the overall quality of a performance by assigning a single numerical value to represent a specific category of accomplishment. They are used for measuring both products and processes.

True-False items

require less time to construct but are most prone to guessing and are prone to a variety of other issues

focus

should be concerned with as a single issue

Psychomotor Domain

•*It is the most difficult to classify into taxonomic levels since all but the simplest reflex actions involve cognitive, and often affective, components. •Includes virtually all behaviors •Best for lower elementary or movement focused courses (nonverbal communication).

RESTRICTED response item

•Most commonly used •Limits answer to a page of less •Typically used to measure comprehension, application, and analysis

Taxonomy of Educational Objectives

•Refers to the cognitive, mental, or thought complexity called for by the objective. •The levels are assumed to be hierarchical. That is, higher-level objectives are assumed to include, and be dependent on, lower-level cognitive skills. •Each level is described using action verbs that are indicative of the different levels

EXTENDED response item

•Requires more than a page in length •Appropriate for term papers and end-of-semester reports •Typically used to measure synthesis and evaluation

ESSAY item

•The student "supplies" the answer rather that "selects" a response •The length may vary •The complexity of the response may vary

Completion items (fill in the blank)

•rival true-false items in ease of construction. •Since answers must be supplied, they are least subject to guessing •require more scoring time