Table 9. Categorical Concurrence Between Standards and Assessment as Rated by Six Reviewers Florida Grade 9 athematics Number of Assessment Items - Standards evel by Objective Hits Cat. Goals Objs # of objs by % w/in std Title evel ean S.D. Concurr. # # evel by evel A - Number Sense, Concepts, and Operations 5.7 6 50 0. YES A - Number Sense, Concepts, and Operations 5.7 0. YES A - Number Sense, Concepts, and Operations 5.7 6 0. YES B - easurement 8.7 0.67.60 YES B - easurement 8.7 7 87 0.67.60 YES C - Geometry and Spatial Sense 5 0.8.0 YES C - Geometry and Spatial Sense 5 60.8.0 YES D - Algebraic Thinking 75 7.8.67 YES D - Algebraic Thinking 5 7.8.67 YES E - Data Analysis and Probability 7 9.7. YES E - Data Analysis and Probability 7 57 9.7. YES Total 7 6. 7 9 50.5 5.09 Total 7 6. 9 5 50.5 5.09 Total 7 6. 0 7 50.5 5.09
Table 9. Depth-of-Knowledge Consistency Between Standards and Assessment as Rated by Six Reviewers Florida Grade 9 athematics Number of Assessment Items - Title Standards Goals # Objs # Hits evel of Item w.r.t. Standard % Under % At % Above S.D. S.D. S.D. S.D. DOK Consistency A - Number Sense, Concepts, and Operations 5.7 0. 6 5 6 YES B - easurement 8.7 0.67.60 9 7 67 9 5 0 YES C - Geometry and Spatial Sense 5.8.0 5 0 7 0 0 0 WEAK D - Algebraic Thinking 7.8.67 69 9 8 NO E - Data Analysis and Probability 7 9.7. 76 7 6 7 NO Total 7 6. 50.5 5.09 9 5 6 5 8
Table 9. Range-of-Knowledge Correspondence and Balance of Representation Between Standards and Assessment as Rated by Six Reviewers Florida Grade 9 athematics Number of Assessment Items - Title Standards Goals # Objs # Hits Range of Objectives % of # Objs Hit Total Balance Index Rng. of Know. % Hits in Std/Ttl Index Hits ean S.D. ean S.D. ean S.D. ean S.D. ean S.D. Bal. of Represent. A - Number Sense, Concepts, and Operations 5.7 0..8. 0 NO 0 6 0.79 0.07 YES B - easurement 8.7 0.67.60 0.75 5 9 YES 0.79 0.07 YES C - Geometry and Spatial Sense 5.8.0.7 0.7 6 7 YES 5 0.8 0.07 YES D - Algebraic Thinking 7.8.67.8 0.69 7 7 YES 5 0.8 0.06 YES E - Data Analysis and Probability 7 9.7..5 0.5 50 7 YES 8 0.90 0.07 YES Total 7 6. 50.5 5.09.7.09 55 5 0 5 0.8 0.08
Table 9. Summary of Attainment of Acceptable Alignment evel on Four Content Focus Criteria as Rated by Six Reviewers Florida Grade 9 athematics Number of Assessment Items - Standards Categorical Concurrence Alignment Criteria Depth-of- Knowledge Consistency Range of Knowledge Balance of Representation A - Number Sense, Concepts, and Operations YES YES NO YES B - easurement YES YES YES YES C - Geometry and Spatial Sense YES WEAK YES YES D - Algebraic Thinking YES NO YES YES E - Data Analysis and Probability YES NO YES YES
Table 9.5 Source-of-Challenge Issues by Reviewer Florida Grade 9 athematics Item Number Comments by Reviewer #5 gives the formula needed to solve this problem. 5
Table 9.6 Depth-of-Knowledge evels by Item and Reviewers Intraclass Correlation Florida Grade 9 athematics Item Rater Rater Rater Rater Rater 5 Rater 6 5 6 7 8 9 0 5 6 7 8 9 0 5 6 7 8 9 0 5 6 7 8 9 0 6
Table 9.6 Depth-of-Knowledge evels by Item and Reviewers Intraclass Correlation Florida Grade 9 athematics Item Rater Rater Rater Rater Rater 5 Rater 6 5 6 7 8 9 50 7
Table 9.7 Notes by Reviewer Florida Grade 9 athematics Item Number Comments by Reviewer 7 too simple 7 Too easy. Doesn't fit No area benchmark. 6 This could be a if you consider the number of steps. Too easy for 9th grade. measurement is content. not sure of matching benchmark straight forward ratios, too easy measurement is content. not sure of matching benchmark. 6 doesn't fit 7 too easy for 9th grade 8 poor question. not sure of matching benchmark 9 too easy for 9th grade this is really a secondary objective because no benchmark fits as the primary objective. 8
Table 9.8 DOK evels and Objectives Coded by Each Reviewer Florida Grade 9 athematics Item DOK0 PObj0 SObj 0 E... E... E..... 5 A... 6 A... 7 D... 8 D... 9 B... 0 A... E... E... E... C... 5 C... 6 B... 7 B... 8 B... 9 B... 0 B... B... B... DOK PObj SObj DOK PObj SObj E... E... E... E... E... D... C.... D... A... B... A... A... A... D... D... A... B... C... C... A... A... B... B.... E... E... E... E... C... C.... C... C..... B... A... B... A... B... B... B... B... C... C... B... B... A.. A... DOK PObj SObj E... E... D... D... D... A... A... A... A.5.. D... D.... C... A... B.... E... E... E... B... C.... B... B... A... A... B... B.... C.... B... B... DOK PObj SObj E... E... D... D... A... A... A.5.. D.... A... A... E... A... B.... C... B... B... B... A... B... C... B.... A... A... DOK5 PObj5 SObj 5 E... E... D... D... B... A... A... D.... A... B.... E... E... E... C... C.... B... B... B... C... B... B... 9
Table 9.8 DOK evels and Objectives Coded by Each Reviewer Florida Grade 9 athematics Item DOK0 PObj0 SObj 0.. 5 D... 6 C... E... 7 E... 8 D... 9 E... 0 A.... A... E... D... 5 B... 6. 7. 8 E... 9 A... 0 B... B... E... D.... D... DOK PObj SObj A... B... D.... E... D... E... A... B... C... A... E... A... B... B... C.... E... A.... D... B... D.... D... DOK PObj SObj DOK PObj SObj. B.... B... D... D... C... C... E... E... D... D... E... E... A... A... C... C.... A... A... A... E... E... B... B... B... B... B... C...... E... E... A... B... A... C... C.... B.. D... D... D... D..... D... D... DOK PObj SObj A... A... B.... D... D... E... E... D... E... A.... A... E... A... B... B.... E... A... C... B... A.... D... DOK5 PObj5 SObj 5 D... B... D... C... E... D... E... A... C... A... A... E... A... D... C.... E... A... C... B... D.... D... 0
Table 9.8 DOK evels and Objectives Coded by Each Reviewer Florida Grade 9 athematics Item DOK0 PObj0 SObj 0 5 6 7 8 9 50 DOK PObj SObj DOK PObj SObj DOK PObj SObj DOK PObj SObj DOK5 PObj5 SObj 5
Table 9.9 Objectives Coded to Each Item by Reviewers Florida Grade 9 athematics ow {} ow- edium {} edium {} edium- High {H} High {H} 0 >0 - <6.06 6.06 >6.06 - <9 9 {} E.. E.. E.. E.. E.. E.. {} E.. E.. E.. E.. E.. E.. {H} D.. D.. D.. D.. D.. E.. E.. {H} C.. D.. D.. D.. D.. 5 {} A.. A.. A.. A.. B.. B.. 6 {H} A.. A.. A.. A... A... A... A... A.. 7 {} A.. A.5. A.5. D.. D.. D.. 8 {H} A.. B.. D.. D.. D.. D.. D.. 9 {H} B.. C.. C.. C.. 0 {} A... A... A... A... A... A.. {H} A.. B.. B.. B.. B.. E.. {} E.. E.. E.. E.. E.. E.. {H} A.. E.. E.. E.. E.. E.. E.. E.. {H} B.. B.. C.. C.. C.. C.. 5 {H} C.. C.. C.. C.. C.. C.. 6 {} B.. B.. B.. 7 {H} A.. A.. B.. B.. B.. B.. B.. B.. 8 {H} A.. A.. A.. B.. B.. B.. B.. B.. B.. 9 {} B.. B.. B.. B.. B.. B..
Table 9.9 Objectives Coded to Each Item by Reviewers Florida Grade 9 athematics 0 {H} B.. C.. C.. C.. C.. C.. {H} B.. B.. B.. B.. B.. {H} A.. A.. A.. A.. B.. B.. B.. B.. {H} A.. A.. A.. B.. D.. {H} B.. B.. B.. B.. 5 {} D.. D.. D.. D.. D.. D.. 6 {H} C.. C.. C.. C.. D.. E.. 7 {H} E.. E.. E.. E.. E.. E.. E.. 8 {} D.. D.. D.. D.. D.. D.. 9 {} E.. E.. E.. E.. E.. E.. 0 {} A.. A.. A.. A.. A.. A.. {H} B.. C.. C.. C.. C.. {H} A.. A.. A.. A.. A.. A.. A.. A.. {} E.. E.. E.. E.. E.. E.. {H} A.. A.. A.. B.. B.. B.. D.. 5 {H} B.. B.. B.. B.. B.. B.. D.. 6 {} B.. C.. C.. C.. 7 {} 8 {} E.. E.. E.. E.. E.. E.. 9 {H} A.. A.. A.. A.. A.. A.. B.. 0 {H} B.. C.. C.. C.. C..
Table 9.9 Objectives Coded to Each Item by Reviewers Florida Grade 9 athematics {H} {H} {} {} 5 6 7 8 9 50 B.. A.. D.. B.. D.. D.. B.. D.. D.. B.. D.. D.. B.. D.. D.. D.. D.. D.. D.. D.. E..
Table 9.0 Items Coded by Reviewers to Each Objective Florida Grade 9 athematics ow ow edium edium edium- High High {} {} {} {H} {H} 0 >0 - < 5.559 5.559 >5.559 - < A A.. A.. A.. 6 6 6 {H} A.. {} A... 6 6 6 6 0 0 0 0 0 {H} A.. A.. {} A.. A.. A.. {} A.. A.. 9 9 {} A.. 6 7 8 0 7 7 0 0 0 0 0 0 9 {H} 9 9 9 A.. A.. 5 5 5 5 8 8 8 {H} A.5. A.5. 7 7 {} B B.. B.. 9 6 6 8 9 9 9 9 9 9 5 5 {H} 5 5 5 B.. 7 0 6 0 {} B.. 6 7 7 7 {H} B.. {} 5
Table 9.0 Items Coded by Reviewers to Each Objective Florida Grade 9 athematics B.. {} B.. 7 7 {} B.. B.. 5 5 8 8 8 8 8 8 9 {} B.. B.. B.. 5 {} C C.. C.. 0 6 0 {} C.. C.. 9 9 9 5 5 5 5 5 5 0 0 0 0 6 6 {H} 6 6 6 6 0 0 0 C.. 9 9 9 6 6 6 0 6 {H} 6 0 5 5 6 7 7 7 7 7 7 0 {H} D D.. D.. 7 7 7 8 5 8 8 5 {H} D.. 8 8 {} D.. D.. 5 {} D.. 8 8 5 5 5 5 6 8 8 8 8 {H} E E.. E.. {H} 6 E.. {H} 8 8 8 8 8 8 6
Table 9.0 Items Coded by Reviewers to Each Objective Florida Grade 9 athematics E.. E.. E.. {H} E.. {} E.. E.. E.. 7 7 7 7 7 9 9 9 9 9 7 7 9 7
Table 9. Assessment Item DOK vs Consensus DOK (Item Number: Number of Reviewers [Average DOK]) Florida Grade 9 athematics ow DOK () ow- atched DOK () atched DOK () atched- High DOK (H) High DOK (H) > - < > - < A []: A.. []: A.. []: A.. []: A.. []: A... []: A.. []: A.. []: A.. []: A.. []: A.. []: A.. []: 6: [.] : [] 6: [.5] H : [] : [] :6 [.5] 0:5 [] : [] : [] 8
Table 9. Assessment Item DOK vs Consensus DOK (Item Number: Number of Reviewers [Average DOK]) Florida Grade 9 athematics A.. []: : [] 9: [] A.. []: 6: [] H 7: [] 8: [] H 0: [] H 7: [] H : [] : [.5] H 0:6 [] : [] H : [] H 9: [.5] H A.. []: A.. []: 5: [.75] : [] : [] 8: [.] H A.5. []: A.5. []: 7: [] B []: B.. []: B.. []: 9: [] : [] 6: [] 8: [] H 9:6 [.8] : [] : [] : [] 5:5 [] B.. []: 7: [] 0: [] : [] 6: [] 0: [] B.. []: : [.5] 6: [] 7: [.67] : [.67] : [] : [] : [.] : [] : [] B.. []: : [] B.. []: : [] H : [] H B.. []: 7: [.5] : [] : [] 9
Table 9. Assessment Item DOK vs Consensus DOK (Item Number: Number of Reviewers [Average DOK]) Florida Grade 9 athematics B.. []: B.. []: 5: [] 8: [] 8:5 [] 9: [] B.. []: B.. []: B.. []: 5: [] C []: C.. []: C.. []: 0: [] 6: [] 0: [] C.. []: C.. []: : [] 9: [] : [] 5:6 [.8] 0: [.75] 6: [.5] : [.75] 6: [.5] 0: [] C.. []: []: []: 9: [] : [] : [] 6: [] 0: [] : [] : [] : [.] : [] 6: [] 0: [] []: : [] : [] : [] 5: [] 6: [] 7:6 [.7] 0: [] :6 [.7] 0
Table 9. Assessment Item DOK vs Consensus DOK (Item Number: Number of Reviewers [Average DOK]) Florida Grade 9 athematics D []: D.. []: D.. []: : [] 7: [.] 8: [] : [] 5: [] 8: [] : [] 5: [] : [] :5 [.8] D.. []: : [.5] 8: [] D.. []: D.. []: 5: [] : [] : [.5] : [] D.. []: : [] 8: [] 5: [.5] 6: [] 8: [.5] : [] H E []: E.. []: E.. []: :6 [.5] : [.5] : [] H :6 [.7] : [] 6: [] : [] H E.. []: :5 [.8] :6 [.5] 8:6 [.7] E.. []: E.. []: E.. []: : [] 7:5 [.] 9:5 [.]
Table 9. Assessment Item DOK vs Consensus DOK (Item Number: Number of Reviewers [Average DOK]) Florida Grade 9 athematics E.. []: E.. []: E.. []: E.. []: : [] 7: [.5] 9: []
Table 9. Group Consensus SSS Grade 9 athematics, athematics, Grade 9 evel Description DOK A Number Sense, Concepts, and Operations A.. The student understands the different ways numbers are represented and used in the real world. A.. Associates verbal names, written word names, and standard numerals with integers, rational numbers, irrational numbers, real numbers, and complex numbers A.. Understands the relative size of integers, rational numbers, irrational numbers, and real numbers A.. Understands concrete and symbolic representations of real and complex numbers in realworld situations A... Understands that numbers can be represented in a variety of equivalent forms, including integers, fractions, decimals, percents, scientific notation, exponents, radicals, absolute value, and logarithms A.. The student understands number systems. A.. Understands and uses the basic concepts of limits and infinity A.. Understands and uses the real number system A.. Understands the structure of the complex number system A.. The student understands the effects of operations on numbers and the relationships among these operations, selects appropriate operations, and computes for problem solving. A.. Understands and explains the effects of addition, subtraction, multiplication, and division on real numbers, including square roots, exponents, and appropriate inverse relationships A.. Selects and justifies alternative strategies, such as using properties of numbers, including inverse, identity, distributive, associative, and transitive, that allow operational shortcuts for computational procedures in real-world or mathematical problems A.. Adds, subtracts, multiplies, and divides real numbers, including square roots and exponents, using appropriate methods of computing, such as mental mathematics, paper and pencil, and calculator A.. The student uses estimation in problem solving and computation. A.. Uses estimation strategies in complex situations to predict results and to check the reasonableness of results A.5. The student understands and applies theories related to numbers. A.5. Applies special number relationships such as sequences and series to real-world problems B easurement B.. The student measures quantities in the real world and uses the measures to solve problems. B.. Uses concrete and graphic models to derive formulas for finding perimeter, area, surface area, circumference, and volume of two- and three-dimensional shapes, including rectangular solids, cylinders, cones, and pyramids B.. Uses concrete and graphic models to derive formulas for finding rate, distance, time, angle measures, and arc lengths B.. Relates the concepts of measurement to similarity and proportionality in real-world
Table 9. Group Consensus SSS Grade 9 athematics, athematics, Grade 9 evel Description DOK situations B.. The student compares, contrasts, and converts within systems of measurement (both standard/nonstandard and metric/customary). B.. Selects and uses direct (measured) and indirect (not measured) methods of measurement as appropriate B.. Solves real-world problems involving rated measures (miles per hour, feet per second) B.. The student estimates measurements in real-world problem situations. B.. Solves real-world and mathematical problems involving estimates of measurements, including length, time, weight/mass, temperature, money, perimeter, area, and volume and estimates the effects of measurement errors on calculations B.. The student selects and uses appropriate units and instruments for measurement to achieve the degree of precision and accuracy required in real-world situations. B.. Determines the level of accuracy and precision, including absolute and relative errors or tolerance, required in real-world measurement situations B.. Selects and uses appropriate instruments, technology, and techniques to measure quantities in order to achieve specified degrees of accuracy in a problem situation C Geometry and Spatial Sense C.. The student describes, draws, identifies, and analyzes two- and three-dimensional shapes. C.. Uses properties and relationships of geometric shapes to construct formal and informal proofs C.. The student visualizes and illustrates ways in which shapes can be combined, subdivided, and changed. C.. Understands geometric concepts such as perpendicularity, parallelism, tangency, congruency, similarity, reflections, symmetry, and transformations including flips, slides, turns, enlargements, rotations, and fractals C.. Analyzes and applies geometric relationships involving planar cross-sections (the intersection of a plane and a three-dimensional figure The student uses coordinate geometry to locate objects in both two and three dimensions and to describe objects algebraically. Represents and applies geometric properties and relationships to solve real-world and mathematical problems including ratio, proportion, and properties of right triangle trigonometry Using a rectangular coordinate system (graph), applies and algebraically verifies properties of two- and three-dimensional figures, including distance, midpoint, slope, parallelism, and perpendicularity D Algebraic Thinking D.. The student describes, analyzes, and generalizes a wide variety of patterns, relations, and functions. D.. Describes, analyzes, and generalizes relationships, patterns, and functions using words, symbols, variables, tables, and graphs. D.. Determines the impact when changing parameters of given functions D.. The student uses expressions, equations, inequalities, graphs, and formulas to represent
Table 9. Group Consensus SSS Grade 9 athematics, athematics, Grade 9 evel Description DOK and interpret situations. D.. Represents real-world problem situations using finite graphs, matrices, sequences, series, and recursive relations D.. Uses systems of equations and inequalities to solve real-world problems graphically, algebraically, and with matrices E Data Analysis and Probability E.. The student understands and uses the tools of data analysis for managing information. E.. Interprets data that has been collected, organized, and displayed in charts, tables, and plots E.. Calculates measures of central tendency (mean, median, and mode) and dispersion (range, standard deviation, and variance) for complex sets of data and determines the most meaningful measure to describe the data E.. Analyzes real-world data and makes predictions of larger populations by applying formulas to calculate measures of central tendency and dispersion using the sample population data and using appropriate technology, including calculators and computers E.. The student identifies patterns and makes predictions from an orderly display of data using concepts of probability and statistics. E.. Determines probabilities using counting procedures, tables, tree diagrams, and formulas for permutations and combinations E.. Determines the probability for simple and compound events as well as independent and dependent events E.. The student uses statistical methods to make inferences and valid arguments about realworld situations. E.. Designs and performs real-world statistical experiments that involve more than one variable, then analyzes results and reports findings E.. Explains the limitations of using statistical techniques and data in making inferences and valid arguments 5