The task of developing a good assessment tool varies according to it intended purpose(s). Were you to develop a formal, summative, final assessment of a process or product for purposed of making critical decisions about learner progression or competency, you would likely want to take the time and effort to ‘standardize’ the tool by providing hard evidence of its content, criterion-related and construct validity and its reliability. If, on the other hand your goal were to develop an informal classroom assessment that could provide you with a measure of students’ content mastery while simultaneously providing students with good formative feedback to help them judge their progress, then you probably only need to assure that the questions demonstrate content validity and are well-constructed according to basic principles of item writing.
Although critically important, determining the purpose of assessment or evaluation is only one major component involved in the decision-making process. Hemmer and Battistone (2001) recommend that decisions regarding the development and implementation of any evaluation tool or program address the following basic areas/questions:
Once the purpose, goals, techniques and logistics of assessment are chosen, it’s time to prepare the actual instrument or tool. In general, the process involved in developing an assessment tool is similar, regardless of the actual format of the items or response mechanism chosen. Typically this involves three major stages: planning, implementation and evaluation (modified from Cornell University, n.d.).
1) Plan for the assessment
a) Outline the knowledge, skills or product to be assessed
b) Identify the learning outcomes to be measured by the assessment
c) Prepare a blueprint (i.e., ‘test’ matrix, table of specifications, task analysis)
d) Use blueprint to choose/match items consistent with learning outcomes
2) Prepare/Implement the assessment
a) Write items according to rules of construction for the type(s) chosen
b) Select items to be included in the test according to table of specifications
c) review and edit items according to guidelines
d) Arrange items; decide on
i) grouping of items
ii) sequence of items within groups
iii) sequence of groupings
e) Prepare directions; if needed prepare directions for individual items or sections
f) Decide on method of scoring and ‘cut score’
g) Administer the assessment
3) Evaluate and revise the assessment
a) If objective tool used for summative purposes, perform test and item analysis
b) Retain, edit as necessary, or discard items on basis of analysis outcomes
c) Revise as indicated
Cornell University Office of Instructional Support (n.d.). Test construction manual. Ithaca, NY: Cornell University. Available at: http://www.clt.cornell.edu/campus/teach/faculty/Materials/TestConstructionManual.pdf
Hemmer, P.A., and Battistone, M.J. (October, 2001). Evaluation and grading (Precourse). Tucson: Clerkship Directors in Internal Medicine Meeting.