Level of Knowledge in Benchmark-Based Assessment Analysis Gerald Kulm Curtis D. Robert Professor Texas A&M University

Most assessments, whether large-scale or not, use the traditional notion of items aimed a different levels of understanding. This idea was popularized over 30 years ago by Bloom (1956) who described six cognitive levels: Recall, Understanding, Application, Analysis, xxxxx, and Synthesis. These levels were applied extensively in the development of "behavioral objectives" and the idea that learning took place through a progression through the hierarchy of cognitive levels.

In testing, the traditional approach was to use a Content by Process matrix to assure that a test contained an appropriate balance of items addressing content topics and at different levels of knowledge. The Process dimension was typically the latest version of levels of knowledge. Behavioral objectives were used to guide the construction of test items.

In the past 10 or 20 years, cognitive scientists have hypothesized three levels or types of knowledge: Declarative, Procedural, and Strategic (Anderson, 1983; xxxx). These levels are less hierarchical than the previous behaviorist levels, developing through connections and in situated learning contexts. Reflecting this theory, applications of levels of knowledge to assessment nowadays typically include 3 or 4 levels of understanding. For example, the NAEP mathematics test uses Conceptual Understanding, Procedural Knowledge, and Problem Solving to describe each item. TIMSS calls these levels Performance Expectations and uses 4 levels for Math (Knowing, Performing Routine Operations, Using Complex Procedures, Solving Problems) and 5 levels for Science (Understanding Simple Information; Understanding Complex Information; Theorizing, Analyzing, and Solving Problems; Using Tools, Routine Procedures, and Science Processes; and Investigating the Natural World).

In recent test development efforts, there is a movement away from strict matrix models for constructing tests, toward overlapping content and level of knowledge dimensions. Test developers usually acknowledge that a single item or task, especially if it is open-ended, can address more that one level of knowledge or even more than one content area. In some cases, efforts are made to develop sets of items that span more than one content topic and include different levels of knowledge.

Although test developers write items to address their own set of content topics or performance objectives, and also claim they are written at particular levels of understanding, the resulting items may not be aligned with national standards and benchmarks. There have been some recent efforts to develop and implement procedures for judging and alignment of assessments and standards in mathematics and science (Kulm, 1998; Stern, 1999; Webb, 1998, 1999). Kulm and Stern focus mainly on content alignment, leaving the issue of levels of understanding aside, except that content alignment includes the idea of "level of sophistication." This term refers to whether an item is best aligned with a benchmark from a particular grade level band, rather than an earlier or later one. The level of sophistication of a benchmark has some relation to level of understanding. For example, the same idea might be at a TIMSS level of "Understanding Complex Information" at grades 3-5, but "Performing Routine Procedures" at grades 9-12. However, neither Kulm nor Stern address alignment this way. If an item is at an inappropriate grade level, it is simply not aligned with the benchmark at hand.

Webb’s alignment procedure directly addresses the level of understanding issue. Analysts classify the standards or benchmarks that serve as the content criteria using 4 levels:

1. Recall - Recall of a fact, information, or procedure


2. Skill/Concept - Use of information, conceptual knowledge, procedures, two or more steps, etc.


3. Strategic Thinking - Requires reasoning, developing a plan or sequence of steps; has some complexity; more than one possible answer; generally takes less than 10 minutes to do.


4. Extended Thinking - Requires an investigation; time to think and process multiple conditions of the problem or task; and more than 10 minutes to do non-routine manipulations.

The analysts then make a judgment on whether or not an assessment item aligns with a standard or benchmark, thereby placing the item in a content-by-process category. An item may be classified for more than one standard. Webb found great variations across states on the consistency between the level of understanding of standards and the percentage of items that matched these levels. He also found that analysts varied in their interpretations of these levels and probably needed more training in order to make consistent judgments.

If Project 2061 is to include types or levels of knowledge or understanding as part of assessment analysis, where should it look for justification or guidance? One starting place is our own Benchmarks. Learning goal statements provide the best key to the type and of knowledge that is portrayed. Chapter 14 Issues and Language discusses the way Benchmarks made decisions about how to characterize knowledge. The use of benchmarks as criteria for aligning content with assessments will require some adaptation or interpretation. As stated in Benchmarks Chapter 14,

If the major use of benchmarks had been to construct tests, action verbs would have provided the best chance of making the test fit the goal. But this was not the intent. (page 312)

Even though the main intent of benchmarks was not to construct tests, they can be used as alignment criteria. We can look to the language in the benchmarks as a guide to the types of knowledge that are expected. For example, in a choice between "graded words" (be aware, know, understand) with the same knowledge statement and "same words" with graded statements, the project chose the latter for Benchmarks. In practice, the "same words" choice is somewhat more complex – see Figure 1 for the details.

Figure 1.  Knowledge Characterizations in Benchmark Statements

The prefaces to benchmark statements can be interpreted as types of knowledge. In fact, the two levels in Benchmarks prefaces (know that, be able to) correspond directly with Anderson’s (1983) definitions of declarative and procedural knowledge. Declarative knowledge (knowing that) refers to concepts that are stored in schemas along with knowledge about their function, form, and preconditions. The few cases in which "know why" is used as a preface refer to the function of declarative knowledge.

Procedural knowledge (knowing how) is stored as production systems or sets of condition-action pairs. If circumstances match some existing patterns of declarative knowledge, the action is performed. According to the theory, all knowledge initially comes in declarative form (Clements & Battista, 1992). From this perspective, there are two types of knowledge in benchmarks: declarative and procedural.

Chapter 14 indicates that the benchmark statements have "graded targets" of knowledge. These graded targets can be interpreted in terms of levels of knowledge. One way that the statements are graded is in level of complexity. For example, Figure 2, which is reproduced from page 313 of Benchmarks, illustrates this notion of graded targets. In this example, complexity is shown through characteristics such as the level of detail in the three statements, the amount of information provided about the idea, and the technical vocabulary used.

Figure 2.  Graded Knowledge Complexity in Benchmark Statements

Another way in which benchmark statements are graded with respect to level of knowledge is in the use of verbs. In particular, some benchmarks are given as active statements about phenomena, ideas, or relationships. For example,

Some source of "energy" is needed for all organisms to stay alive and grow. (5E, grades 3-5#2)

The operations + and – are inverses of each other—one undoes what the other does; likewise x and −. (9A, grades 6-8#4)

These statements imply that students should know the idea in the sense of giving examples, providing explanations, and applying the knowledge in various situations or problems.

 Copyright© 2000, by Gerald Kulm.  All Rights Reserved.