Does Your Assessment Align with Standards? Mathematics Examples Gerald Kulm Curtis D. Robert Professor of Mathematics Education Texas A&M University

Prepared for the National Council of Teachers of Mathematics regional meeting, Boise, Idaho, October 8, 1999.

In order to determine whether an assessment task is aligned with mathematical ideas contained in a standard, two essential questions should be addressed. First, does the content of the task match the content of the standard? Second, does the format of the task reflect the intention and expectations of instruction? Project 2061 of the American Association for the Advancement of Science is developing a procedure that can be used to analyze K-12 mathematics and science assessments for (1) alignment with a set of benchmarks or standards, and (2) the quality of their presentation (AAAS, 1998). The criteria used in the procedure provide a detailed and comprehensive approach to determining the specific mathematics or science content that is assessed by tasks and instruments, and producing a profile of the quality of format of a task. The term "standards" is used generically to denote the learning goals used as content criteria for analysis. In practice, the standards could be statements from the NCTM Standards (1989, 1998), AAAS Benchmarks (1993), the National Science Education Standards (NRC, 1996), the National Assessment of Educational Progress Framework, or a state or local mathematics or science framework.

The analysis consists of four stages. First, a Preliminary Inspection identifies the benchmarks that are apparently addressed by each task. Then a complete analysis of Content Alignment and the Technical Quality is done for each task. Finally, a summary report and profile is constructed that describes the characteristics of the task.

In this paper, a single mathematics standard and a set of assessment tasks is used to illustrate the Content Alignment component of the analysis procedure. At this stage, a preliminary inspection has already been done and the selected standard appears to be addressed by this set of assessment tasks. The list of a full set of proposed analysis criteria is included in the Appendix.

In doing a Content Alignment analysis, it is essential to (1) specify exactly what mathematical ideas are contained in the standard, (2) clarify the meaning of these ideas by referring to other standards statements and documents, and (3) identify student cognitive development, difficulties, or misconceptions about the ideas, using research findings or teacher experience as a guide. The following summary provides this information for a geometry standard taken from Benchmarks for Science Literacy (AAAS, 1993).

1. Many objects can be described in terms of simple plane figures and solids.
2. Shapes can be compared in terms of concepts such as parallel and perpendicular, congruence and similarity, and symmetry.
3. Symmetry can be found by reflection, turns, or slides.

Idea 1: The term "object" can mean both real, physical objects and mathematical objects such as geometric figures themselves. Ways to describe the objects can include verbal descriptions, drawings or sketches, or physical models.

Idea 2: Shapes can include both two and three-dimensional shapes. The concepts following the phrase " such as" are not inclusive of all possible concepts for comparing shapes, but ought to be included. The terminology itself is less important than understanding the concepts and being able to use them in making comparisons.

Idea 3: The phrase "can be found" includes the expectation of being able to describe the nature of a symmetry with one of these three concepts and being able to identify and produce a symmetric figure that involves one of the three.

The corresponding NCTM Standards 2000 (draft) for the benchmark are statements under the standards for grades 3-5:

• Analyze characteristics and properties of two- and three-dimensional geometric objects
• Recognize the usefulness of transformations and symmetry in analyzing mathematical situations

According to NCTM Standards, describing involves "learning to use mathematical terminology by hearing it used repeatedly in context." In comparing, students tell what the characteristics are of each shape and how they are different.

According to Van Hiele levels, students (1) identify shapes with concrete examples, then (2) become able to use properties to identify and describe shapes. These levels, progress in them, and what students understand about shapes is highly influenced by instruction, which is lacking in the early grades. Given appropriate instruction, students can understand abstract properties of geometric figures by 5^th grade (Clements & Battista, 1992). Students expand their notion of symmetry by 5^th grade using more than one line of symmetry and describing rotational symmetry more precisely with angle measures.

After identifying the ideas in the selected benchmark and clarifying their meaning and substance, the following questions help to guide an analysis of the content alignment between the standard and the assessment task. In order to achieve reliability and consistency in rating these criteria, the analysts use a set of indicators and a scoring guide.

Substance - Does the task address the specific substance of the benchmark or is there only a "topic" match? Does the substance of the alignment depend on student responses? For open-ended or performance items, how well do the intended answers or scoring rubric align with the benchmark?

Sophistication - Does the task reflect the level of sophistication of the benchmark or does it target a standard at an earlier or later grade level?

Beyond Standards - Does the task assess content that is not required for achieving the standard?

Content and Context - Is the assessment context (use of tools, time allowed, response mode) of the task consistent with the content intended by the standard?

Cognitive Demand - Does the knowledge type (procedural, conceptual, application) of the task match that intended or implied by the standard?

The following tasks address one or more of the ideas in the standard. In order to do an analysis, complete information about the task is required, including the conditions under which it is administered, the scoring rubric, and sample student responses.

Example 1: Compare Shapes Task (NAEP, Grade 4, 1996)

Think carefully about the following question. Write a complete answer. You may use drawings, words, and numbers to explain your answer. Be sure to show all of your work.

A. In what ways are the figures above alike? List as many ways as you can.
B. In what ways are the figures above different? List as many ways as you can.

A. The figures are alike because: (Do not accept: They both have lines that are straight)
-They both have 4 sides (or 4 corners or 4 angles)
-They both have parallel sides
-They both have two sets of sides that are the same length
-They have the same area
-They have the same length (base)
-They have the same height
-The both have little squares
4 sides and 4 angles are not considered different reasons

B. The figures are different because: (Do not accept: They’re not both the same shape.)
-One has 4 equal angles and the other does not
-One has right angles or perpendicular lines and the other does not.
(Students don’t need to make the comparison; they can just say one has 4 equal angles)
-One is slantier that the other (or one takes up full squares and the other does not)
-They have different perimeters

1. Correct: They have 4 sides. They have parll sides. Their the same legth. On the top they both have lines that are strait and they both have four corners. Incorrect. They have the same room.
2. Correct: One has square corners. One is more slant. One has slanted lines and the other one doesn’t. Incorrect: They aren’t the same shap. They don’t have the same amount of full cubs.

Substance - The item most closely addresses idea 1 that shapes can be described and idea 2, that shapes can be compared. According to the scoring key, the item includes comparison using the concepts parallel and perpendicular, although full credit (two examples of alike or different) can be obtained without using the ideas of parallel or perpendicular. The match with the idea depends on the student response, so there is not assurance that the task assesses the full idea.

Sophistication - The comparison criteria (parallel, perpendicular, length, height, area) are appropriate to the grade level, since the formal terms are not required. Fourth grade students may have difficulty with this conceptual comparison if their instruction has not included similar work.

Beyond Standards - All of the content knowledge required in the task is included in the standard

Content and Context - The use of diagrams, and squared paper are appropriate for the task at this grade level. Students are expected to compare shapes by visual approaches, and may be aided by the squared units.

Cognitive Demand - The standard requires conceptual understanding of the geometric terms and relationships. The assessment task also requires conceptual understanding, without using the terms themselves. However, a correct answer can be obtained only with a description, not making a comparison, so the cognitive demand of the item is somewhat low for idea 2.

Example 3: Folding Symmetry (TIMSS, grade 4)

Craig folded a piece of paper in half and cut out a shape.

Draw a picture to show what the cut-out shape will look like when it is opened up and flattened out.

Correct Response (Note: See the examples above. The accuracy in drawing is not important, nor is the size of the figure.)

10 The drawing of the cut-out shape corresponds to figure A.
11 The drawing of the remaining piece of paper corresponds to figure B.
19 Other correct.

Incorrect Response

70 Drawing corresponds to figure C.
71 Drawing corresponds to figure D.
72 Drawings correspond to figures E or F or G.
79 Other incorrect.

Substance - The item most closely addresses only the part of idea 3 related to symmetry through reflection. Drawing the shape that shapes requires the student to understand properties of reflection across a line. On the other hand, an inaccurate drawing, which is scored as correct may not reveal a full understanding of the properties of reflection. Also, without a requirement for an explanation, students may be able to produce a satisfactory drawing without a full understanding of symmetry or reflection.

Sophistication - The requirement to draw the complete figure does address the level of sophistication expected by the standard. The task seems to find squarely within the grade 3-5 expectations.

Beyond Standards - All of the content required for answering the question is contained in the standard. Drawing does requires some skills that are not a part of the standard, but the scoring guide indicates that accuracy is not important. There is also some requirement for spatial visualization that may be outside of the standard.

Content and Context - At this grade level making a drawing is appropriate, and sketching one that is reasonably accurate is an appropriate approach. Some students might do better if concrete material (a paper with the figure drawn on it) were available.

Cognitive Demand - The standard is at the conceptual level, and the task is also at this level of cognitive demand. It might be possible that students who have a great deal of experience with similar tasks could produce the drawing without conceptual understanding but not likely, since the figure is not familiar, such as a heart or tree, for example.

Example 3: Turning the Letter P (Minnesota Practice Test, grade 3)

Jeremy traced his cutout of the letter P and then moved the cutout as shown.

What kind of a move did he make with the cutout?

1. a turn
2. a flip
3. a slide

Substance - The item most closely addresses idea 3. On the other hand, the single turn that is pictured does not produce a symmetric figure, so a complete assessment of the substance of this part of the benchmark isn’t attained. The multiple choice format, however, only provides an opportunity to assess if students can identify the appropriate move. It does not include whether students see this as a type of symmetry, or are able to select or produce an image using a turn move. And, of course, a correct answer could be obtained by guessing or by knowing two of the concepts, eliminating them to get the answer. If a student misses the question, it cannot be assumed that he/she doesn’t understand the other two types of symmetries.

Sophistication - The items appears to be at the right level, however, the terms "turn," and " flip" may confuse students who have learned these as rotations and reflections.

Beyond Standards - All of the content knowledge necessary for answering the question is included in the standard.

Content and Context - The context for the task is neutral, involving a typical setting for a paper and pencil task. However, for some third grade students, a physical cutout may produce more valid information about their knowledge.

Cognitive Demand - The item does require conceptual understanding of the properties of a turn symmetry, although at a fairly low recognition level. A full understanding of a turn symmetry requires more complete knowledge than assessed in this task.

Example 4: Compare Square (NAEP 1996, grade 4)

Laura was asked to choose 1 of the 3 shapes N, P, and Q that is different from the other 2. Laura chose shape N. Explain how shape N is different from shapes P and Q.

N is a square (but P and Q are triangles.)
OR N has four sides, (or vertices or points or angles), (but P and Q each have 3).
OR All the sides of N are =, (but not so with P and Q)
OR All the angles of N are = (or, all right angles, (not so with P and Q)
Must mention both P and Q or neither P nor Q plus N is a square or give other characteristics of N.

Has four sides. (correct)
N is not round anywhere on the letter. (incorrect)
N shape is square and the other two are rectangles. (incorrect)

Analysis of Content Match (Example 4): (An exercise for the reader)

Substance -

Sophistication -

Beyond Standards -

Content and Context -

Cognitive Demand -

American Association for the Advancement of Science. 1993. Benchmarks for Science Literacy. New York: Oxford University Press.

American Association for the Advancement of Science (1998). A proposal to develop resources for science and mathematics literacy: assessment. Washington, DC: Author.

Clements, D. H. & Battista, M. T. (1992). Geometry and spatial reasoning. In D. Grouws (Ed.), Handbook of research on mathematics teaching and learning. New York: Macmillan.

National Assessment of Educational Progress (1996). 1996 Assessment mathematics public release grade 4. www.ed.gov/nces/naep.

National Council of Teachers of Mathematics. (1989). Curriculum and Evaluation Standards for School Mathematics . Reston, VA: Author, 1989.

National Council of Teachers of Mathematics. (1998). Standards for school mathematics: Standards 2000 (Draft) . Reston, VA: Author, 1989.

National Research Council. (1996). National science education standards. Washington, DC: National Academy of Sciences Press.

Third International Mathematics and Science Study (1996). TIMSS mathematics items: Released set for population 1 (3 ^rd and 4^th grade). www.timss.org.

Copyright© 2000, by Gerald Kulm.  All Rights Reserved.