A) In most cases, a test will be one part of a fully developed assessment plan. Most programs have cognitive, attitudinal, and performance goals. Tests are commonly used in association with cognitive goals – to review student achievement with respect to a common body of knowledge associated with a discipline or practice. Tools such as interviews and surveys are used more commonly in conjunction with other non-cognitive assessment techniques.

A) Use of a test is a good choice when:

• the student is to demonstrate acquisition of knowledge or ability to process and use knowledge (i.e., the outcome of interest is cognitive in nature).
• the student's knowledge about a wide range of content is to be evaluated (for example, survey and capstone courses).
• multiple observations of the content-related knowledge are needed (for example, math and foreign languages).
• more resources are available for constructing the assessment instrument, rather than for scoring and reporting results.
• a large group is being assessed.

A) In comparison to other assessment procedures, tests can sample student knowledge with efficiency and reliability – you can find out what a lot of students know in a brief period of time. A second advantage is that repeated use of a test will provide a means of comparison between different student groups or the same group over time. This type of testing practice provides reviewers with a rich context for evaluation, decision-making, and recommendations.

A) At a minimum, a good test will have:

• a well-defined purpose or intent.
• a foundation based on a set of written goals and objectives.
• some evidence that the test purpose was achieved – reliability and validity information about test items, the test as a whole, and the relationship between test scores and other indices of academic performance.
• detailed scoring procedures that allow for specific interpretations and feedback to those tested and those making decisions.

A) One common practice is to develop and adopt common pretests and posttests in courses with multiple sections. Items for common tests could be culled from existing exams. Another practice is to determine a portion of each unit exam, a specific set of items, that are scored for program assessment and also for individual evaluation. This practice is sometimes referred to as course-embedded testing. (When using course-embedded testing, it is important to notify students how the practice affects the assignment of grades.)

A) Generally, instructors develop their own classroom tests, making all decisions about when and how to construct, administer, score, and report results of tests. Construction is often done without formality or documentation. With assessment, planning, implementing, and using results become a group effort – a shared set of decisions and responsibilities. Consensus is emphasized. Some additional planning time, communication, and record keeping will be needed. The most frequent use of tests by instructors is to assign grades related to individual student learning. When used for program assessment, test performance is generally used along with other information to describe group achievement and is independent of grading.

A) Because tests for assessment are different than tests for grading, it is recommended that you provide a brief orientation which covers the following topics:

• the purpose of the test and how results will be used
• how the students will benefit
• whether guessing is appropriate
• whether names and identification numbers are needed and why

Q) What is a standardized test?

A) A standardized test is one in which the initial construction, as well as conditions for administration and scoring, have a uniform procedure so that the scores may be interpreted in a consistent manner from one administration to the next. These are designed by test development specialists, either internally or externally.

A) Seven basic and sequential steps are recommended:

1. Determine the outcomes to be measured.
2. Develop a test blueprint.
3. Write the test items.
4. Review, critique, and edit the items.
5. Pilot the items.
6. Obtain reliability and validity data.
7. Revise, reuse, and report.

Q) What are the differences between a locally-developed and an externally-developed test?

A) Differences between locally- and externally-developed tests are described below.

Q) Are there any important decisions that precede test development?

A) Yes. Before a test is developed, a few decisions about use of results need to be clarified because they will have an influence on test construction. Three critical decisions are raised here.

• Are group or individual results needed? Good questions to ask are:
• Can individual scores be created from group scores?
• Are individual-level data necessary for programmatic decisions?
• Do the decisions being made affect individuals?
• What types of item formats are needed? Items can be broadly classified as either recognition or production types. A detailed comparison of types of tests is provided in Table 1.  Good questions to ask are:
• Does the item format allow inferences about the thinking skills of interest?
• Is there a simpler format for the information of interest?
• Are criterion- or norm-referenced results needed? Good questions to ask follow brief descriptions of each type of test.

Criterion-referenced tests:

• provide absolute comparisons.
• describe the desired student performance directly and in detail.
• are used to judge performance by some pre-set standard (proficiency or mastery).


Norm-referenced tests:

• provide relative comparisons.
• describe performance in terms of the relative position held by some known group or individual.
• are used to judge performance in comparison to others.


Good questions to ask are:

• Is mastery of information or thinking skill necessary to move to a new area of study?
• Do all students need to show a minimal level of knowledge? By a certain time?
• Did students begin with a wide range of ability?

Tests that are soundly constructed are built to meet specifications just as a house is built according to a plan. The plan for a test is frequently called a test blueprint (also called a table of specifications or a test matrix). Two dimensions are laid out as the columns and rows of a two-way table. At the intersections between the rows and columns, the number of items needed to test the area (or percentage of importance) is recorded. A simple example is provided in Figure 1.

Things to notice in the blueprint shown in Figure 1:

• The distribution of items by content areas is relatively equal although there is a slightly greater emphasis on Revolution–36% versus 32% in each of the other two areas.
• The process that appears to be most important is Recall of Facts–64% versus 30% or 6% of the items.
• Because of the fairly equal distribution of emphasis on three content areas and the heavy emphasis on knowledge of factual information, this test might be used early in a term as a review or as a pretest with new learners.

Content areas, thinking processes, and importance specifications are given in test blueprints. Using the steps below and the blank test blueprint (Exercise 1) a practice test blueprint can be drafted.

Steps for Completing a Test Blueprint

1. First, identify the discipline/topic area to be addressed by the test in the blank provided.
2. Next, define the test's content areas. Along the left column, list at least 3 general content areas from your discipline.
3. Now, define the intellectual processes to be assessed. Along the top row, list at least three intellectual processes.
4. Finally, define the importance of the test content areas and processes to the overall test. Enter the number and/or percentage of items in the blanks where content areas and processes overlap. The values should reflect the importance of content/process relationships as conveyed in the curriculum and instruction.

Suggestions for Developing Test Blueprints

• Refer back to the goals and objectives written for the assessment plan.
• Assign priorities to goals and objectives so that the most important content areas and thinking skills are addressed by your test.
• Remember that a test provides a sample or samples of behaviors and should be combined with other information about performance.

Functions of a Test Blueprint – A Summary:

• Helps achieve balance between instruction and assessment.
• Reduces tendency to test "memory of facts" only.
• Helps ensure that a test will sample all important content and process areas.
• Provides a structure for communicating with students before and after testing.
• Provides a structure for analyzing, summarizing, and reporting results.

Before you begin writing items for a test, use the following checklists to consider possible resources.

Internal:

• unit tests or finals from individual sections of a course
• departmental tests – retired and current
• student-generated items – however, the items usually need revision
• resource staff and material centers (i.e., assessment office, library, exam services)

External:

• Curriculum materials that accompany textbooks may provide unit/chapter tests and item banks.
• Professional agencies associated with a discipline may have retired tests or item banks related to certification or licensing exams.
• Colleagues teaching at other institutions may have developed instruments/items for similar needs.
• Test companies sell published instruments and item banks.
• Journals that focus on teaching of the discipline sometimes contain unpublished tests. Social science and educational journals that focus on instructional practice are good sources.

Suggestions for Reviewing an Externally-Constructed Test

• Remember that it is unlikely there will be 100% match between an externally-constructed test and local curriculum and instructional practice.
• Before adoption, all test need to be examined in terms of:
• the match between the purpose for which the test was designed and the local needs based on stated student learning goals and objectives
• the correspondence between the test blueprint and content of related items
• Review the technical merits of the instrument. Look and/or ask for the following:
• norms/technical manual
• test blueprint
• information about item development
• evidence of reliability and validity
• relatively current norms, information about norm groups
• Pilot the instrument with a small group before commitment to full-scale use.
• Call other users – publishers should provide a list of contacts.
• Always check costs before use: usable/non-reusable materials, scoring (local or external), reports.

This section contains general item writing tips as well as specific tips for various item types.

Suggestions for All Item Types

• Write items directly after instruction when possible.
• Write more items than are needed – 25% more is a good rule of thumb.
• Keep reading and vocabulary levels simple unless you are testing those skills (i.e., avoid superfluous wording).

• Make sure the problems posed in the item stems are clear and unambiguous.
• Use "concrete" situations and pictorial, graphic, or tabular stimuli when possible.
• Use novel material in formulating problems to measure higher level mental processes (i.e., analysis and application type items); however, guard against overuse of this strategy.
• Avoid lifting statements verbatim from a text unless your intention is to test memorization skill or knowledge for simple and basic information.
• Avoid single and double negative items; phrase in a positive format if possible.
• Be careful of clues and answer cues within the item stem.

• Avoid noun modifiers like "all", "never", "no", and "always"
• Order the response options in some parallel way (e.g., alphabetically, chronologically), but avoid creating response patterns (e.g., an alternating true-false pattern, overuse of the "C" option in a set of multiple-choice items).

Suggestions/Checks for Multiple-Choice Items

• Avoid highly technical response options.
• Avoid having the correct answer longer than the incorrect options.
• Use responses that are plausible and homogeneous in some way.
• Use between three and five options. Try to use the same number of options for all items, however do not create superfluous options just to maintain a parallel format.

Suggestions/Checks for Matching Items

• Provide more choices than number of statements to be answered unless a choice can be used more than once.
• Have students choose answers from the column with the least amount of reading.

Suggestions/Checks for True-False (Alternative Response) Items

• Avoid ambiguous and indefinite terms of degree or amount (i.e., "frequently", "in most cases", etc.).
• Avoid negative and double negative statements.
• Keep true and false statements approximately the same length.
• Have approximately the same number of true and false items on a test.

Suggestions/Checks for Writing Completion or Short Answer Items

• Draft items that require a single-word answer or a brief and definitive statement.
• Avoid statements that may be logically answered by several terms.
• Indicate the unit of expression (i.e., date, percentage) when answers require numerical information.

Suggestions/Checks for Writing Essay Items

• Select items carefully because of the limited number that can be given in a single time frame.
• Make items clear and specific so that scoring can be done easily.
• Establish a framework within which the student will write
• Limit the area covered by an item.
• Indicate the value of items and suggest time parameters.
• Decide, in advance, the factors considered in evaluation and note them in the instructions.

Analyzing the quality of test items can be accomplished in phases. Some general suggestions follow:

• Before the test is used (pilot tested), ask faculty members to check for the following:
• Congruence between test content and test blueprint
• Do items reflect the specified topic weights?
• Do items measure the range/emphasis of cognitive abilities specified?
• Clarity of directions
• Use of good item construction practices
• Accuracy, specificity, and emphasis of scoring/rating procedures
• After a test is given for the first time (pilot tested) refer to the following checklists:

If using an objective test format, check for:

• Item difficulty
• Item discrimination

If using other formats, check for:

• Consistency of application of scoring criteria by those doing the rating
• Have a sample of responses scored by a second evaluator.
• Have a sample of responses scored in a different order and at a different time

Before test scores are interpreted or used for decisions, it is essential to review how well the items functioned. Calculating item difficulty and discrimination values is recommended practice.

An item’s difficulty index is expressed as the proportion of students who responded correctly (successfully) to an item. If scores from all students in a group are included the difficulty index is simply the total percent correct. When there is a sufficient number of scores available (i.e., 100 or more) difficulty indexes are calculated using scores from the top and bottom 27 percent of the group.

The value is interpreted in an inverse way, that is, a high value is interpreted as less difficult. These values can range from 0 to 1.00 and are usually expressed as a percentage. Item difficulty is calculated in the following way:

Item difficulty is generally interpreted in the following way:

Item: Who is the current host of the Tonight Show?

1. Johnny Carson
2. Arsenio Hall
3. David Letterman
4. Jay Leno

An item’s discrimination index is expressed as the difference among high and low scorers. The value is interpreted in terms of both direction (positive or negative) and strength (non-discriminating to strongly-discriminating). These values can range from -1.00 to +1.00.  Item discrimination is calculated in the following way:

Item discrimination is generally interpreted in the following way:

The pattern of incorrect responses is another feature of item analysis. Ideally, each incorrect response should be selected by a higher percentage of low- than high-scoring students. In the previous example, responses A and B are selected by few students; these options do not provide much information about student knowledge and should be revised.

The relationships of difficulty and discrimination patterns to intended test purpose is a critical aspect of item analysis. The table below outlines how the majority of items should function for norm- and criterion-referenced tests.

Analyzing the quality of a test involves an examination of validity and reliability characteristics of the test. Some basic definitions and critical concerns are reviewed below.

Validity:

• is the most important quality of a test.
• does not refer to the test itself.
• generally addresses the question: "Does the test measure what it is intended to measure?"
• refers to the appropriateness, meaningfulness, and usefulness of the specific inferences that can be made from test scores.
• is the extent to which test scores allow decision makers to infer how well students have attained program objectives.

Reliability:

• is the degree to which test scores are free of errors of measurement due to things like student fatigue, item sampling, student guessing.
• of objective tests is threatened by poor item construction.
• of other test formats is threatened by rater or scorer variability and inconsistent application of scoring/rating standards.

For Further Reading

Berk, Robert. (Ed.). Performance Assessment: Methods and Applications. Baltimore, MD: The Johns Hopkins University Press, 1986.

Ebel, Robert. and Frisbie, D. A. Essentials of Educational Measurement (4th ed.).  Englewood Cliffs, NJ: Prentice-Hall, 1986.

Gronlund, Norman. Measurement and Evaluation in Teaching (4th ed.). New York: MacMillan, 1991.

Gronlund, Norman. Preparing Criterion-Referenced Tests for Classroom Instruction.   New York: MacMillan, 1973.

Hill, John. Measurement and Evaluation in the Classroom (2nd. ed.). Columbus,OH: Charles E. Merrill, 1981.

Kubiszyn, Tom and Borich, G. Educational Testing and Measurement: A Guide for Writing and Evaluating Test Items. Minneapolis, MN: Burgess Publishing Co., 1984.

McKeachie, Wilbert. Teaching Tips: A Guidebook for the Beginning College Teacher (7th ed.). Lexington, MA: D.C. Heath & Co., 1978.

Mehrens, William. and Lehmann, I. J. Measurement and Evaluation in Education and Psychology. (2nd ed.). New York: Holt, Rinehart, & Winston, 1978.

Office of Examination, Scanning, & Evaluation Services. "Interpretive Recommendations for Item and Test Analysis Computer Output", 8/13/85.

Osterlind, Steven. Constructing Test Items. Boston: Kluwer Academic Press, 1989.

Plake, Barbara. (Ed.) Social and Technical Issues in Testing: Implications for Test Construction and Usage. Hillsdale, NJ: Erlbaum, 1984.

Priestley Michael. Performance Assessment in Education and Training: Alternative Techniques. Englewood Cliffs, NJ: Educational Technology Pub., 1982.

Popham, W. J. Criterion-Referenced Measurement. Englewood Cliffs, NJ: Prentice-Hall.

Thorndike, Robert. and Hagen, E. Measurement and Evaluation in Psychology and Education. (4th ed.). New York: John Wiley, 1978.

Tyler, Ralph. and Wolf, R. M. Crucial Issues in Testing. Berkeley, CA: McCutchan, 1974.

Wergin, Jan. "Basic Issues and Principles in Classroom Assessment." In J.H. McMillan (ed.), Assessing Students' Learning. San Francisco: Jossey-Bass [New Directions for Teaching and Learning Series (no. 34)], 1988.