REF 469 Test 2 Review
1. Clearly indicate cognitive skills or process that should be employed to formulate the answer, 2. Clarify the pupil's task with concise details 3. Set appropriate page or time limits.
*A WELL-CONSTRUCTED ESSAY ITEM should...
•DEBATES •MOCK TRIALS •GIVING A CLASS A PRESENTATION •REENACTMENTS •SCIENCE EXPERIMENTS •MUSICAL PERFORMANCES
*Examples of Performance Assessments:
performance-based
*Teachers establish situations that allow them to observe and rate learners directly as they analyze, problem solve, experiment, make decisions, measure, cooperate with others, present orally, or produce a product in what type of test
they use DIRECT measures of learning rather than indicators
*The MAJOR ADVANTAGE of performance tests is that
Taxonomy
*The __________ of educational objectives for the cognitive domain helps categorize objectives at different levels of complexity.
•to test a student's ability to use higher level cognitive skills/mental processes •When objective test items are inappropriate •Few test items are necessary •Test security is in question
*What are ESSAY items best used for?
Instructional objective
*____________ should be stated in observables, behavioral terms (using an action verb), in order for two or more individuals to agree that a student has or has not displayed the learning outcome in question.
Multiple choice items
*can measure behavior at higher levels of taxonomy (ex: comprehension, synthesis, evaluation) but are most difficult of the objective items to construct and should be used with caution on younger children
The affective taxonomy
*describes objectives that reflect underlying emotions, feelings, or values, rather than cognitive or thought complexity.
CHECKLISTS
2 VARIATIONS/characteristics in student behavior are being observed - it's either PRESENT or ABSENT. They are best suited for complex behaviors or performances that can be divided into a series of clearly defined, specific actions.
•Can assess complex learning objectives •EASY to construct •Can assess communication skills •They eliminate student guessing
ADVANTAGES of essay items
•What is the goal of your test? Link directly to objectives •Know the odds of students passing by guessing •*Objective Test items should also be objective in scoring
Advantages and disadvantages of the different formats are (these should influence choice):
Conveys to the teacher the... •# of items to be constructed per objective •level of taxonomy for each item •*Whether the test represents a balanced picture of the objectives that were taught
Advantages of using a test blueprint
Students have to develop positive dispositions and "habits of mind" which include behaviors such as: •Constructive criticism •Tolerance of ambiguity •Respect for reason •Appreciation for the significance of the past
Affective domain and social domain
SPECIAL CONDITIONS
An instructional objective describes any ___________ in which the learning will take place.
1. Be STUDENT DIRECTED (students will be able to...) 2. Contain an OBSERVABLE BEHAVIOR or an ACTION VERB specifying the learning outcome 3. State any SPECIAL CONDITIONS under which the behavior must be displayed 4. State a PERFORMANCE LEVEL considered to be indicative of mastery - it must be MEASURABLE
An instructional objective should (four steps):
1. What kinds of essential tasks, achievements, or other valued competencies am I missing with paper-and-pencil tests? 2. What accomplishments of those who practice my discipline are valued but left unmeasured by conventional tests? •Skills related to acquiring information •Skills related to organizing and using information
Cognitive domain - ask the following questions to guide selection of objectives
•Absolutes in wording (i.e. always, all, never, only) •Double negatives•Requiring and opinion •Double-barreled statements (i.e., use of and, or) •Excessive wordiness •A tendency to reflect statements taken verbatim from readings (w/o context, or out of context)
Common pitfalls of true-false items
Percentages •# of items correct •# of consecutive items correct or consecutive errorless performances •Essential features included •Completion within a prescribed time limit •Completion with a certain degree of accuracy
Criterion levels may be stated as
•Takes longer to score/grade •Scoring can be unreliable-essay tests have lowest "INTER-SCORER RELIABILITY" •Essays sample limited content •Answers are susceptible to bluffing
DISADVANTAGES of essay items
Practicing the violin; Practicing multiplication tables; Watching the evening news
EXAMPLES of learning activities
•*Objectives specify higher-level cognitive processes •When objective test items such as T/F or Multiple Choice are inappropriate •Few tests or few test items are necessary •*Test security is a question
Essay items should be used when...
In the field of athletics, diving and gymnastics are examples of performances that judges rate DIRECTLY; at band contests, judges DIRECTLY see, hear, and rate students as they perform
Example of a performance-based test
GIVEN A CALCULATOR, students will be able to multiply two-digit numbers, correct to the nearest whole number
Example of stating conditions
Be familiar with the law; Appreciate art deco; Enjoy speaking French
Examples of UNOBSERVABLE LEARNING OUTCOMES
Given 20 two-digit addition problems, the student will be able to compute at least 18 OUT OF 20 answers correctly. Given 20 two-digit addition problems, the student will be able to compute at least 90% OF THE PROBLEMS correctly.
Examples of criterion levels
Fixing a car radio; Adding signed numbers correctly; Punctuating an essay correctly
Examples of learning outcomes
Circle the initial sound of words; Recite the names of the characters in Tom Sawyer; Change the spark plugs on an engine
Examples of observable learning outcomes
1. Deciding WHAT to test 2. Designing the assessment context 3. Specifying the scoring rubrics
First 3 steps in constructing a performance assessment
focus, brevity, and clarity
For objective items, consider:
•*Use pictorial, graphical or tabular stimuli (Pictures, drawings, graphs, tables, maps and so on require the student to think at the application level of Taxonomy of Educational Objectives and may involve higher cognitive processes.; Use of these stimuli can generate several items at this level) •Use analogies that demonstrate relationships among terms •Require the application of previously learned principles or procedures to novel situations
Higher level multiple choice questions
Create a list of objectives that specifies the knowledge, skills, habits of mind, and indicators of the outcomes that will be the focus of your instruction.•
How to decide what to test (step 1)
Instructional objective
If the observable learning outcome is to take place at a particular time, in a particular place, with particular tools or materials, etc., this must be stated in the
instructional objectives, instructional activities, and test items.
In the 3-stage model of classroom measurement to enhance CONTENT VALIDITY we need alignment between
•Questions that use tabular data, charts, and graphs that come from the text as well as newspapers, magazines, and reports related to the curriculum •Require students to apply think about and apply information that comes from real-life situations
OPEN-BOOK questions and exams are ideal for...
cognitive, affective (attitudes, social skills), or psychomotor processes •assessments of processes, products, or both.
Performance tests can measure
•Too many blanks •Lack of specificity (too many potential responses) •Failure to state a problem
Pitfalls of completion items
•True-False items •Multiple Choice items •Completion items
Possible formats include
•*Grammatical cues or specific determiners 9ex. The use of "a" or "an") •Multiple defensible answers •Unordered option lists •Stem clues •Opinionated statements •Lack of a problem statement in the stem •Redundant wording •Wordiness in the correct option
Possible pitfalls of multiple choice include:
This is a technique used to help instructors remember to write objectives and test items at differentcognitive levels (of Bloom's Taxonomy) to ensure that tests include items that measure higher-level thinking.
Purpose of the test blueprint
brevity
Should be as succinct without sacrificing...
1.What does the doing of ___ look and feel like to professionals? 2.What projects and tasks are performed by these professionals? 3.What roles - or habits of mind - do professionals assume?
Some questions to ask when designing the assessment context are
A particular time; A particular place; Special equipment that is used, tools, or other resources
Special conditions may include
criterion levels
Stating the __________ is the part of the instructional objective that indicates HOW WELL the behavior is to be performed; MEASURABLE part of the objective
•ID cognitive processes you want student to use BEFORE writing the test item •*Clearly state the task including grading criteria in writing and orally •Avoid beginning essay items with what, who, when, and list unless you are only measuring at knowledge level •Ask for evidence/support for a controversial position •Avoid using optional items (have everyone answer all essay items-no picking and choosing) •Establish reasonable time and/or page limits •Use essay Q's for measuring learning outcomes that cannot be adequately measured by objective items (T/F, multiple choice, matching...) •Be sure item matches instructional objective
Suggestions for WRITING essay items
•Write good essay items •Using Restricted Range (time/page limits) whenever appropriate •Using an ADVANCED RUBRIC or PREDETERMINED SCORING SCHEME •Consistent scoring with ALL students •GRADE ANONYMOUSLY by removing or covering names to avoid bias •Scoring ALL student responses to ONE ITEM before moving to the next one •Keep scores from previous items hidden when scoring other items •Rescoring papers before returning them and averaging discrepant scores
Suggestions for more objective/reliable SCORING of essay items...
1.If possible, items should require a single-word answer/brief and definite statement. 2.Be sure the question/statement poses a problem to the examinee. •A direct question is often more desirable than an incomplete statement. 3.The answer required is factually correct. •Watch the language is precise and accurate in relation to the subject matter 4.Omit only key words •Don't eliminate so many elements that the sense of the content is impaired. 5.*Write the blank as near the end of the sentence rather than near the beginning. •This will prevent awkward sentences. 6.If the problem requires a numerical answer, indicate the units in which it is to be expressed
Suggestions for writing completion or supply items
1.The stem of the item should clearly formulate a problem. •Include only material to make the problem clear/specific •Include as much of the item as possible •keep the response items short 2.*Be sure that there is one and only one correct/clearly best answer 3.Be sure wrong answers choices are plausible. •Eliminate unintentional grammatical clues •Keep the length and form of all the answer choices equal •Rotate the position of the correct answer from item to item randomly 4.Use negative questions or statements only if the knowledge being tested requires it •It is more important for students to know what it is rather than what is not 5.Include from 3-5 options (2-4 distractors w/ the correct answer) •optimize testing for knowledge (not) encouraging guessing. •Not necessary to have the same number of answers for each item 6.*To increase the difficulty of a multiple-choice item, increase the similarity or content among the options 7.Use the option "none of the above" sparingly and only when the keyed answer can be unequivocally as right or wrong. 8.Avoid using "all of the above". It is usually the correct answer and makes the item too easy for students with partial information.
Suggestions for writing multiple choice items
1.Very basic - instructions as to how to mark T/F should be clearly explained prior to the test 2.*Make certain statements are definitely either true or false (not opinion) (focus) •If using opinion, attribute it to a specific source 3.Use relatively short statements and eliminate extra material (brevity) 4.Keep statements about the same length 5.Use approximately equal numbers of true and false items 6.Avoid using double-negative statements (clarity)
Suggestions for writing true-false items
Performance-based tests
Tests that are often used in science, social studies, or math
Clarity
The meaning of the item should be crystal clear to all test-takers. The goal is to reduce error in the measurement process.
•writing appropriate questions that elicit complex cognitive skills •structuring the student's response so that other factors do not obscure your ability to evaluate it. •(ex. Writing skills of student)
The potential of the essay question as an evaluation device depends on
1.What knowledge or content is essential for learner understanding of the subject matter? 2.What intellectual skills are necessary for the learner to use this knowledge or content? 3.What habits of mind are important for the learner to successfully perform with this knowledge or content?
Three general questions to ask when deciding what to teach
Restricted response item and extended response item
Two main types of essay items
OBSERVABLE
Two or more individuals observing a student should be able to agree that the student demonstrated the desired learning outcome, therefore, instructional objectives must be __________
•Hands on exercise •Problems to solve •Observable outcomes or product •A process that can be observed
What to provide when designing the assessment context
Learning outcomes
_________ are ends (products); implies a specific product or result
LEARNING ACTIVITES
__________ are the means (processes) to the ends; only implies an activity that will lead to a learning outcome
Instructional
_____________ objectives should include the level of proficiency to be demonstrated and the special conditions under which the skills must be demonstrated.
Performance test
______________ is being advocated as a fairer and more relevant alternative to conventional testing.
Instructional Objective
a clear and concise statement of the skill or skills (outcomes) that your students will be expected to perform after a unit of instruction.
RUBRICS
are CRITERIA FOR GRADING/SCORING performance tests. They are samples of acceptable responses against which the rater compares a student's performance.
Rating scales
assign numbers to categories representing different degrees of performance. They are typically used for those aspects of a complex performance such as: attitudes, products, and social skills, that do not lend themselves to yes/no or present/absent type judgments.
Holistic scoring
estimates the overall quality of a performance by assigning a single numerical value to represent a specific category of accomplishment. They are used for measuring both products and processes.
True-False items
require less time to construct but are most prone to guessing and are prone to a variety of other issues
focus
should be concerned with as a single issue
Psychomotor Domain
•*It is the most difficult to classify into taxonomic levels since all but the simplest reflex actions involve cognitive, and often affective, components. •Includes virtually all behaviors •Best for lower elementary or movement focused courses (nonverbal communication).
RESTRICTED response item
•Most commonly used •Limits answer to a page of less •Typically used to measure comprehension, application, and analysis
Taxonomy of Educational Objectives
•Refers to the cognitive, mental, or thought complexity called for by the objective. •The levels are assumed to be hierarchical. That is, higher-level objectives are assumed to include, and be dependent on, lower-level cognitive skills. •Each level is described using action verbs that are indicative of the different levels
EXTENDED response item
•Requires more than a page in length •Appropriate for term papers and end-of-semester reports •Typically used to measure synthesis and evaluation
ESSAY item
•The student "supplies" the answer rather that "selects" a response •The length may vary •The complexity of the response may vary
Completion items (fill in the blank)
•rival true-false items in ease of construction. •Since answers must be supplied, they are least subject to guessing •require more scoring time