Unit 7: Assessment Methods

Administering and Scoring the Test

-Conditions should be as favorable as possible -written directions should be clear -blackboard illustration for younger students -create a punch out template for correct answers

3- Preparing the Test Specifications

-General outcomes and specific terms have been developed -should not relate to specific course content -writing of test items should be guided by a carefully prepared set of test specifications 1- describe the achievement domain being measure 2-link the achievement domains to course subject matter 3-provide guidelines for obtaining a representative sample of test tasks Building a table of Specifications: 1- selecting the learning outcomes to be tested 2-outlining the subject matter 3- making a two way chart (describes the sample of items to be included in the test)

Simple Item-Analysis Procedure

-Tally of student scores on master copy -1/3 highest and lowest scoring 1-determine the percentage of high scorers and low scoreers - H = 8 out of 10 = 80 -L = 4 out of 10= 40 2- Obtain item difficulty by adding the percentage correct in the high and low groups and dividing by 2. Add a percent sign to the answer 80+40 / 2 = 60% 3-Obtain discriminating power by subtracting the percentage correct in the low group from the percentage correct in the high group. Add a decimal point to the answer. 80-40= .40 -difficulty index is based on the high and low scoring groups only, but this provides a satisfactory approximation of item difficulty -discrimination index typically uses the decimal point and thus ranges from 0 to 1.00

2- Defining the General Outcomes in Specific Terms

-What specific types of student performance that are to be accepted as evidence that the outcomes have been achieved EXAMPLE: what specific types of performance will show that a student "knows the meaning of common terms" or "understands the relevant principles of testing"? -they are observable responses that can be called forth by test items. -What is meant by defining learning outcomes in specific performance terms (key terms: identifies, distinguishes between, selects, describes, matches, explains, predicts, and formulates.) -vague terms should be avoided "learns" "sees" "realizes"- do not clearly indicate performance to be measured -Action VERBS should be used to clarify

Matching Items

-a variation of the multiple choice form. Switch to the matching format only when it becomes apparent that the same alternatives are being repeated in several multiple-choice items -HELPS TO ELIMINATE THE REPITITION OF THE ALTERNATIVE ANSWERS AND PRESENTTHE SAME ITEMS IN MORE COMPACT FORM

Simulated Performance

-an attempt to match the performance in a real situation entirely or partially. EXAMPLE: -swinging bat at imaginary ball -shadowboxing -swimming/tennis strokes -vocational/science- simulate portions of the actual job -

student project

-approach involves a combination of academic, communication, and thinking skills in solving unstructured real-world problems, the construction of a unique project or both. -multiple outcomes and criteria CHARACTERISTICS OF A GOOD STUDENT PROJECT 1- it focuses on multiple learning outcomes 2-includes the integration of understanding, skills, and strategies 3-concerned with problems and activities that relate to out-of-school life 4-active participation of students in all phases of the project 5-provides for student self-assessment and independent learning 6-requires performance skills that are generalize-able to similar situations 7-feasible within the constraints of the students' present knowledge, time limits, and available resources and equipment 8-it is both challenging and motivating to students 9-it is fair and doable by all students 10-provides for collaboration between the students and the teacher

Improving the Functioning Content of Items

-barriers to avoid: -unnecessarily difficult vocabulary -unnecessarily complex sentence structure -statements containing ambiguity -unclear pictorial materials -vague directions -material reflecting race, ethnic, or sex bias AVOID these barriers by: - writing each test item so that it presents a clearly formulated task -stating the items in simple, clear language -keeping the items free from biased and nonfunctional material -using a test format and directions that contribute to effective test taking -common clues to be avoided during test preparation are: -verbal associations that give away the answer -grammatical inconsistencies that eliminate wrong answers -specific determiners that make certain answers probable and others improbable: -stereotyped or textbook phrasing of correct answers -length or location of correct answers -material in an item that aids in answering

Not Performance related

-compare and contrast two speeches from two different political candidates, and the techniques used within them -define the term "persuasive technique" -analyze a speech to determine use of persuasive techniques

Performance Related

-create a commercial that uses persuasive techniques -present a persuasive speech that includes two persuasive techniques

Establishing Criteria and Standards

-criteria for each major area (problem selection, research skills, report writing, product design, and construction, oral presentation) -level of thinking and general problem solving skills - criteria should be checked against instructional objectives -students should choose own problem but may need help from teacher to do so

Paper-and-Pencil Performance

-different from traditional pencil/paper tests by placing greater emphasis on the application of knowledge and skill in a simulated setting EXAMPLE: -may serve as experience before the actual physical performance/expensive equipment, complicated, demonstrating competence first to prevent accidents -"construct"..weather map, short story, floor plan, plan an experiment

rubrics are helpful for setting clear expectations, and for helping teachers and students to have conversations about scores that were given. A possible disadvantage is ______________.

-difficulty in distinguishing between scores and categories *although rubrics are useful, it is sometimes difficult to determine the score or category into which the student's performance should fall For example, there may be a "gray area" where the student's work is on the line between B and B.

Selecting the Focus of the Assessment

-focus on procedure, product, or combination of both.

The Interpretive Exercise

-more effectively measure complex learning outcomes -may be a paragraph, table, chart, graph, map, or a picture. EXAMPLE-attached

Work Sample

-performed under controlled conditions -building-completing all steps that would be required in a real life situation -job skills duplicated in a school setting

Structured Performance Test

-provides for an assessment under standard, controlled conditions. -Making prescribed measurements, adjusting a microscope, following safety procedures iin starting a machine, or locating a malfunction in electronic equipment 1-nature and purpose of test 2-equipment and tools provided 3-testing procedure: ---type and condition of equipment ---description of required performance ---time limits and other conditions 4- method of judging performance

Analyzing the Effectiveness of Test Items

-review test item by item when handed back/discuss results. -why did they select that item? -review of the course material is beneficial

scoring rubrics

-scoring guidelines that describe the characteristics of the different levels of performance used in scoring or judging a performance

Preparing Clear Directions

-should be simple and concise 1-purpose of the test 2-time allowed to complete the test 3-how to record the answers 4-whether to guess when in doubt about an answer

Making the Two Way Chart

-table of specifications- a two way chart that specifies the number of proportion of test items (or assessment tasks) to be designed for each area of content and each type of intended learning outcome when planning a test (or other assessment procedure) -will the test measure the intended learning outcomes and the the major subject matter is included?

true/false items

-typically used to measure the ability to identify whether statements of fact are correct. -yes/no; agree/disagree; right/wrong; fact/opinion (only two responses are possible) -Is statement t or f and if it is false the student should make that statement true; the part that may need changed is underlined

Specifying the Performance Outcomes

-use verbs such as identify, construct, and demonstrate (and their synonyms) -includes a job or task analysis to identify the specific factors that are most crucial in the performance -necessary to obtain a representative sample of the most crucial ones -keep list at a reasonable length EXAMPLE: Demonstrates Skill in Oral Reporting 1- Stands in a natural manner. 2- Maintains good eye contact. 3- Uses appropriate facial expressions 4-Uses gestures effectively. 5-Speaks clearly and with good volume 6-speaks at an appropriate rate 7-presents ideas in an organized manner. 8-Uses appropriate language. 9-Maintains interest of the group. EXAMPLE: Designs and conducts an experiment. Writes an accurate account of the study. States valid conclusions. Writes a critique of the procedure and findings. Presents and defends the study in class.

Identification Test

-used in complex performances "identify/locate a short in the electrical circuit" "identify an unknown substance by identifying procedures and instruments needed to conduct experiment"

Learning Check:

1- Include only homogenous material in each matching item. 2- Keep lists of items short and place the brief responses on the right- hand side 3- Use larger or smaller number of responses than premises. 4- Place responses in alphabetical or numerical order 5- Specify in the directions the basis for matching and state that each response can be used once, more than once, or not at all. 6- Put all matching items on the same page

Rules for Writing True/False Items

1- Include only one central idea in each statement 2-Keep the statement short and use simple vocabulary and sentence structure 3-word the statement so precisely that it can unequivocally be judged tru or false 4-use negative statements sparingly and avoid double negatives 5-Statements of opinion should be attributed to some source unless used to distinguish facts from opinion 6-when cause/effect relationships are being measured, use only true 7-Avoid extraneous clues to the answer 8- Base items on introductory material to measure more complex learning

Preparing Performance Assessments

1- Specifying performance outcomes 2-selecting the focus of the assessment (procedure, product, both) 3-Selecting an appropriate degree of realism 4-selecting the performance situation 5-selecting the method of observing, recording, and scoring.

Rules for Scoring Essay Answers

1- evaluate answers to essay questions in terms of the learning outcomes being measured 2-score restricted-response answers by the point method, using a model answer as a guide 3-grade extended response answers by the rating method, using defined criteria as a guide 4-evaluate all of the students' answers to one question before proceeding to the next question 5-Evaluate answers to essay question without knowing the identity of the writer 6-whenever possible, have two or more persons grade each answer

Selecting the Learning Outcomes to Be Tested

1- recall knowledge 2-intellectual abilities and skills 3-general skills (laboratory performance, communication, work study) 4-attitudes, interests, and appreciations -Acievement testing is most useful when covered in the first two areas of the cognitive domain -In other areas they are evaluated by rating scales, checklists, anecdotal records, inventories, and similar contest assessment procedures -specific types of performance for the test will aid in the construction of the test items that are most relevant to the intended learning outcomes

Rules for Writing Multiple-Choice Items

1-Design each item to measure an important learning outcome 2-present a single, clearly formulated problem in the stem of the item 3-State the stem of the item in simple, clear language 4-Put as much of the wording as possible in the stem of the item 5-state the stem of the item in positive form, wherever possible 6-Emphasize negative wording whenever it is used in the stem of an item 7-Make certain that the intended answer is correct or clearly best 8-Make all alternatives grammatically consistent with the stem of the item and parallel in form 9-Avoid verbal clues that might enable students to select the correct answer or to eliminate an incorrect alternative -----Similarity of wording in both the stem and the correct answer is one of the most obvious clues -----Stating the correct answer in textbook language or stereotyped phraseology may cause students to select it because it looks better than the other alternatives -----Stating the correct answer in greater detail may provide a clue -----Including absolute terms in the distractors enables students to eliminate them as possible answers because such items "always", "never", "all", "none", "only", are commonly associated with false statements -----Including two responses that are all inclusive makes it possible to eliminate the other alternatives, since one of the two must obviously be the correct answer -----Including two responses that have the same meaning makes it possible to eliminate them as potential answers 10-Make the distracters plausible and attractive to the uninformed ----TO increase plausibility: A-Use the common misconceptions or errors of students as distracters B-State the alternatives in the language of the student C-Use "good-sounding" words ("accurate", "important") in the distracters as well as in the correct answer D-Make the distracters similar to the correct answer in both length and complexity of wording E-Use extraneous clues in the distractors, such as stereotyped phrasing, scientific-sounding answers, and verbal associations with the stem of the item. But don't overuse these clues to the point where they become ineffective F-Make the alternatives homogenous, but in doing so beware of fine discriminations that are educationally insignificant 11- Vary the relative length of the correct answer to eliminate length as a clue 12-Avoid using the alternative. "all of the above" and us "none of the above" with extreme caution 13-Vary the position of the correct answer in a random manner 14-Control the difficulty of the item either by varying the problem in the stem or by changing the alternative 15-Make certain each item is independent of the other items in the test 16-Use an efficient item format 17-follow the normal rules of grammar 18-break (or bend) any of these rules if it will improve the effectiveness of the item

What a Student Portfolio Can Show

1-Learning progress over time 2-student's current best work 3-comparison of best work to past work 4-development of self-assessment skills 5-development of reflective learning 6-individual's level and pace of work 7-clear evidence of learning to parents and others 8-the amount of teacher-students collaboration involved

Rules for Constructing Interpretive Exercises

1-Select introductory material that is relevant to the learning outcomes to be measured 2-select introductory material that is new to the examinees 3-keep the introductory material brief and readable 4-construct test items that call forth the type of performance specified in the learning outcome 5-follow the rules of effective item writing that pertain the type of objective

Creation of Learning Objectives Review

1-Specifying the instructional objectives 2-Preparing the test specifications 3-Constructing relevant test items 4-Arranging the items in the test 5-Preparing clear directions 6-Reviewing and evaluating the assembled test 7-Administering the test and making an item analysis

Arranging the Items in the Test

1-Usually desirable to group together items that measure the same outcome 2-where possible, all items of the same type should be grouped together(so that there is one set of directions for each type ) 3- the items should be arranged in terms of increasing difficulty (easy items first to keep students from getting "bogged down")

Writing Selection Items: True/False

1-When there are no possible alternatives, a shift can be made to a t/f item 2-when there are a number of similar factors to be related, a shift can be made to a matching item 3-when the items are to measure analysis, interpretation, and other complex outcomes, a shift can be made to interpretive exercise

Determining the Number of Test Items to be used

1-age of the students tested -time available for testing -type of test items used -type of interpretation to be made -Elementary Students: no more than 20-30 minutes -highschool/college- can take tests for several hours but are usually 40-50 minutes (a class period)

planning for the use of a portfolio

1-define the purpose of the portfolio 2-select the types of entries to include 3-develop methods to evaluate the entries 4-establish means to maintain and use the portfolio 5-evaluate the portfolio

Common Outline for Problem-Solving

1-establishing criteria and standards 2-selecting and stating the problem 3-locating and selecting resources 4-writing the report 5-designing and completing a research study of making a product 6-orally presenting and defending the project

Prepare the student

1-in clear and concise language, explain what the student is expected to do. 2-explain and show what tools, equipment, instruments, and other resources will be available to the student 3- explain how performance will be assessed 4-explain when and how feedback will be provided

Rules for Writing Matching Items

1-include only homogenous material in each matching item 22-keep the lists of items short and place the brief responses on the right 3-use a larger, or smaller, number of responses than premises, and permit the responses to be use more than once. 4-place the responses in alphabetical or numerical order 5-specify in the directions the basis for matching and indicate that each response may be used once, more than once, or not at all 6-put all of the matching item on the same page

evaluating the student's final level of performance

1-prepare a list of criteria for each scoring rubric to be prepared 2-select the number of categories of performance to be used 3-adapt scoring rubrics from published sources

General Guidelines for Item Writing

1-select the type of test item that measures the intended learning outcome most directly 2-write the test item so that the performance it elicits matches the performance in the learning task 3-write the test item so that the test task is clear and definite 4-write the test item so that it is free from nonfunctional material 5-write the test item so that irrelevant factors do not prevent an informed student from responding correctly 6-write the test item so that irrelevant clues do not enable the uninformed student to respond correctly 7- write the test item so that the difficulty level matches the intent of the learning outcome, the age group to be tested, and the use to be made of the results 8-write the test item so that there is no disagreement concerning the answer 9-write the test items far enough in advance that they can be later reviewed and modified as needed 10-write more test items than called for by the test plan- allows you to discard weak or inappropriate items during item review

Assessing the Product

Assessment should be focused on the product 1-different procedures can result in an equally good product (writing a theme) 2-the procedure is not available for observation (take home work) 3-the procedural steps have been mastered 4-the product has qualities that can be clearly identified and judged

1-Specifying the Instructional Objective:

At the end of this unit in achievement test planning the student will demonstrate that he or she... 1-Remembers the meaning of common terms. 2-Remembers specific facts. 3-Remembers the basic procedures. 4-Understands the relevant principles. 5-Applies the principles. (Note that these statements of general learning outcomes are drawn directly from Bloom's Cognitive Taxonomy and have been deliberately kept free of specific course content so that with only slight modification they can be used with various units of study)

Best Practice for Electronic Portfolios

Best Practice- 1-the student helps to select the entries, then reflects on their quality and evaluates the portfolio as a whole 2-the portfolio should contain samples of the student's best work, or work that shows growth 3-having criteria in place ahead of time helps in determining what should be placed in the portfolio, and how it will be assesssed -4portfolios can be used both by the teacher alone and in parent teacher conferences, to show parents what students have been learning 5-student growth or improvement should be measured in a portfolio for any content area NOT best practice- -1 When scoring portfolios, use on or more checklists to evaluate the portfolio in a variety of ways -2- teacher selects work to go in portfolio

Best practice for true/false statements

DO: -keep the statement short and use simple vocab -word the statement so it can be clearly judged true or false -attribute opinions to their source, unless the question focuses on distinguishing fact from opinion -in cause/effect statements, use only true propositions -avoid extraneous clues to the answer DON'T: -If including more than one central idea, clearly identify each one in the statement -strive for a mix of positive and negative statements

Best Practices in Scoring Essays

DO: -score restricted response answers using a point method, with a model answer as a guide -when possible, have two or more persons grade each answer -evaluate answers to essay question in terms of the learning outcomes being measured DON'T: -evaluate all of one students essay answers before going on to the next student's responses -keep each student's strengths and weaknesses in mind when grading

Best practices for multiple choice questions

Do: -Phrase the stem to avoid equivocal answers -use clear, simple language -emphasize negative wording when it is used -use the same level of detail for all distractors -make the distractors plausible -make all alternatives grammatically consistent with the stem Don't -State the stem in negative form when possible -limit the wording in a stem -state the correct answer in textbook language -use "always" and "never" for emphasis -include two responses that have the same meaning

Assessing the Procedure

Focus the performance assessment on the procedure when: 1-there is no product, or product evaluation is feasible (unavailable or too costly) 2-The procedure is orderly and directly observable 3-correct procedure is crucial to later success 4-analysis of procedural steps can aid in improving a product

Skill execution v. authentic assessment

Performing Skill: -Write a short story on a given topic -construct a bar graph from a given set of data -set up laboratory equipment and follow the steps of an assigned exercise Authentic Assessment: -Plan a short story, outline the plot and characters, write the story, review and critique the story, and rewrite it. -analyze a set of data and select the best type of graph to illustrate it/ construct the graph and give 3 reasons for selecting this type of graph -design an experiment for comparing two products, complete the experiment, and write a summary of the findings. Make recommendations concerning the two products and give reasons

Types of Essay Question

Restricted Response: -places strict limits on the answer to be given -usually narrowly defined by the problem and the form of the answer is also commonly indicated by words such as: list, define, and give reason. Sometimes limited further by special directions Extended Response: -gives students freedom to determine the form and scope of their responses.

Essay Questions

Rules: 1- use essay question to measure complex learning outcomes onle 2-relate the questions as directly as possible to the learning outcomes being measured 3-formulate question that present a clear task to be performed 4- do not permit a choice of questions unless the learning outcome requires it 5-provide ample time for answering and suggest a time limit on each question

Short Answer Items

Rules: 1-state the item so that only a single, brief answer is possible 2-start with a direct question and switch to an incomplete statement only when greater conciseness is possible by doing so 3- it is best to leave only one blank, and it should relate to the main point of the statement 4-place the blanks at the end of the statement 5-avoid extraneous clues to the answer 6- for numerical answers, indicate the degree of precision expected and the units in which they are to be expressed

Selecting the Performance Situation

Similar to the degree of realism present in the situation 1- paper/pencil performance 2-identification test 3-structured performance test 4-simulated performance 5-work sample 6- student project Useful in describing and illustrating the various approaches used in performance

Common Standards for Judging Performancs

Type: Rate- -Solve ten addition problems in two minutes -type 40 wpm Error- -no more than two errors per typed page -count to 20 in Spanish Without Error Time: -set up laboratory equipment in five minutes -locate an equipment malfunction in three minutes Precision: -measure a line within one-eighth of an inch -read a thermometer within two-tenths of a degree Quantity- -Complete 20 laboratory experiments -locate 15 relevant references Quality (rating) -write a neat, well-spaced business letter -demonstrate correct form in driving Percentage Correct- -solve 85% of the math problems -spell correctly 90% of the words in the word list Steps Required- -diagnose a motor malfunction in five steps -locate a computer error using proper sequence of steps use of material- -build a bookcase with less than 10% waste -cut out a dress pattern with less than 10% waste Safety- -Check all safety guards before operating machine -drive automobile without breaking any safety rules

anecdotal record

a brief description of some significant event; typically includes the observed behavior, the setting in which it occurred, and a separate interpretation of the event 1-they focus on meaningful incidents 2-they are recorded soon after the incident 3-they contain enough information to be understandable later 4-the observed incident and its interpretation are kept separate

checklist

a list of measurable dimensions of a performance or product, record "yes/no" judgement STEPS: 1-list the procedural steps or product characteristics to be evaluated 2-add common errors to the list, if such is useful in diagnosing poor performance 3-arrange the list in some logical order (sequence of steps) 4-provide instructions and a place for checking each item 5-add a place for comments at the bottom of the form, if needed EXAMPLE: 1-Removes thermometer from container by grasping the nonbulb end 2-wipes thermometer downward from nonbulb end with fresh wiper

Advantages and Disadvantages of rubrics

advantages- teachers provide clear and straightforward expectations from the beginning of the project -rubrics provide expectations on how students can earn a specific score -students are able to justify themselves to the teacher based on the language in the rubric with regards ti what was earned disadvantages- -the "shades of grey" are always an issue- what students have done well and what they are still lacking -it can be a challenge to determine everything that the student captured within a completed project -in assessing the growth of students and giving them credit when needed can be difficult to achieve

holistic scoring

based on an overall impression of the performance or product rather than a consideration of the individual elements

interpretive

efficient means of measuring the interpretation of printed information, but questions are subjective and can include extraneous clues

short answer

guessing is likely, but the responses are time consuming to score

multiple choice

highly structures, and clear tasks are provided. However, finding good distractors can be difficult.

Portfolios

http://jfmueller.faculty.noctrl.edu/toolbox/portfolios.htm

The text notes that each item should be independent of the other items in the test. This means that:

information given in one item should not help the student answer another

In an interpretive exercise, a series of test questions is based on a common section of:

introductory material

When constructing the correct answer for a multiple choice question, what should be varied?

length and position

performance based

measures complex learning that cannot be evaluated via paper and pencil tests, although scoring is often for individuals rather than groups

essay

measures the highest levels of learning outcomes, but scoring can be impacted by writing skill, bluffing, grammar, and handwriting

In a performance assessment, the teacher may assess the procedure, the product, or both. Presenting a persuasive speech would require assessing the _______________.

procedure

Creating a commercial that uses persuasive techniques would require assessing the ___________.

product

Selecting the Types of Test Items to Use

selection-type-presents students with a set of possible responses from which they are to select the most appropriate answer Example: 1-Multiple choice 2- True/False 3-Matching 4-Interpretive exercise -students can get some correct by guessing -can be scored quickly and objectively -eliminate bluffing -eliminate the influence of writing skill -provide an extensive sample of student performance (bc a large number of items are used) -they provide for the identification of specific learning errors supply-type- requires students to create and supply their own answers. Example: 1-short answer 2-essay (restricted response) 3-essay (extended response) -easier to construct but more difficult to score -can be contaminated by answers of varying degrees of correctness and by adjustments needed for misspellings-limited sample of learning tasks that can be measured -restricted to measuring factual knowledge -better suited to measure complex learning outcomes

matching

short reading and response time, but also restricted to simple knowledge outcomes

rating scale-

similar to a checklist and serves the same purpose in judging procedures and products

Nature of Multiple Choice Items

stem- presents a problem situation and several alternatives (options or choices) that provide possible solutions to the problem -the stem may be a question or incomplete statement/ answers include distracters and correct answers

Outlining the Subject Matter

the content of a course may be outlined in detail tor teaching purposes, but for test planning only the major categories need be listed

analytic scoring

used when judgement on each criterion the performance or project is to be scored

true/false

useful when there are only two possible alternatives, but there is no way to know that the student knows the correct information