ACED Q Matrix
Evidence Models
All of the ACED tasks have a single observable outcome variable that is scored dichotomously (correct or incorrect). For this reason each evidence model corresponds to a unique row pattern in the \(Q\)-Matrix. These are replicated according to the number of tasks that use that evidence model for scoring (i.e., the number of tasks built from the task model corresponding to the evidence model).
| Task/Evidence Model | Solve Geometric Problems | Common Ratio | Example | Explicit | Extend | Model | Recursive | Table | Verbal | Visual | Design Pattern | Anchor |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Solve Geometric Problems | ++ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | Compensatory | SGP |
| Common Ratio | 0 | ++ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | Comp | Common Ratio |
| Examples Geometric | 0 | 0 | ++ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | Comp | Examples |
| Explicit Geometric | 0 | 0 | 0 | ++ | 0 | 0 | 0 | 0 | 0 | 0 | Comp | Explicit |
| Extend Geometric | 0 | 0 | 0 | 0 | ++ | 0 | 0 | 0 | 0 | 0 | Comp | Extend |
| Model Extend Table Geometric | 0 | 0 | 0 | 0 | + | ++ | 0 | + | 0 | 0 | Comp | Model |
| Recursive Rule Geometric | 0 | 0 | 0 | 0 | 0 | 0 | ++ | 0 | 0 | 0 | Comp | Recursive |
| Table Extend Geometric | 0 | 0 | 0 | 0 | + | 0 | 0 | ++ | 0 | 0 | Comp | Table |
| Verbal Rule Explicit Model Geometric | 0 | 0 | 0 | + | 0 | + | 0 | 0 | ++ | 0 | Comp | Verbal |
| Verbal Rule Extend Model Geometric | 0 | 0 | 0 | 0 | + | + | 0 | 0 | ++ | 0 | Comp | Verbal |
| Verbal Rule Model Geometric | 0 | 0 | 0 | 0 | 0 | + | 0 | 0 | ++ | 0 | Comp | Verbal |
| Visual Explicit Verbal Rule Model Geometric | 0 | 0 | 0 | + | 0 | + | 0 | 0 | + | ++ | Comp | Visual |
| Visual Extend Geometric | 0 | 0 | 0 | 0 | + | 0 | 0 | 0 | 0 | ++ | Comp | Visual |
| Visual Extend Table Model Verbal RuleGeometric | 0 | 0 | 0 | 0 | + | + | 0 | + | + | ++ | Comp | Visual |
| Visual Extend Verbal Rule Model Geometric | 0 | 0 | 0 | 0 | + | + | 0 | 0 | + | ++ | Comp | Visual |
The + indicates that the corresponding proficiency variable (names are abbreviated) is relevant, the ++ indicates that the variable is the primary one for this task type.
The names of the task and evidence models are created by removing the spaces from the names, making the first letter lower case, and appending the work ‘Task’ (for task models) or ‘Evidence’ (for evidence models).
The design pattern of ‘Compensatory’ means that the experts thought the skills entered in a compensatory manner.
The anchor sets are constrained to have an average difficulty of 0. The anchor column indicates which set this evidence model type belongs to.
The ‘’Solve Geometric Problems’’ tasks are actually tasks about distinguishing geometric from other types of sequences, so the proficiency variable is not correctly names.
Difficulty Targets and Link Model Priors
The ACED team decided to build three different variants of of each type of task: an easy, a medium difficulty and a hard task. To model this, we used a variant of the evidence model called the link model prior, a sub-model that differs from its evidence model parent by having a different prior distribution for the parameters. In this case “easy” tasks were thought to have a difficulty of around -1 (on an IRT scale); “hard” tasks were thought to have a difficulty of around 1; and “medium” tasks were thought to have a difficulty of around 0.
Notationally, the Link Model Prior names were formed from the Evidence Model names by adding ‘Hard’, ‘Medium’, or ‘Easy’, and ‘LMP’. The task specifications (a sub-model of the task model) names were formed by adding just the name. The task names ended in a number indicating the difficulty target (1=easy, 2=medium, 3=hard).
The “Solve Geometric Problems” tasks
In the initial design of ACED, the the team identified one class of problems as “Solve Geometric Problems” tasks and their Q-Matrix entry indicated that they tapped the “Solve Geometric Problems” proficiency model (the highest level one). In later analysis, these tasks proved to be highly influential. On closer scrutiny, it turns out that the tasks involved recognizing whether or not a sequence was geometric. Thus, maybe the proficiency model was really missing a “Recognize Sequence Type” node, which is the one this should be attached to. The original model was the one used for task selection and analysis in all of the papers.
(Note that the CBAL Mathematics Team at ETS, noted the prevalence of a pattern in which students would try to solve proportional reasoning problems using addition instead of multiplication. It may be the sequence which typically forced the recognize sequence problems first, helped students later. Thoughts?)
The complete task list
ACED used an extended Task ID system (designed for StatShop) that used a URL-like syntax to indicate the ECD models associated with a task. The extended IDs look like:
ecd://“Project”/“Assessment”/“Model”/“Submodel”/“ID”.“Version”
In this case “Project” is always K12Math, “Assessment” is always Formative and “Version” is always 1. The “ID” part is unique, always starts with a ‘t’ and ends with a number indicating difficulty. This shorter ID is used in a number of places.
The complete list of tasks is as follows:
ecd://K12Math/Formative/commonRatioTask/commonRatioMedium/tCommonRatio2b.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsMedium/tSolveGeometricProblems2a.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsMedium/tSolveGeometricProblems2b.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsEasy/tSolveGeometricProblems1a.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsEasy/tSolveGeometricProblems1b.1
ecd://K12Math/Formative/visualExtendTableModelVerbalRuleGeometricTask/visualExtendTableModelVerbalRuleGeometricMedium/tVisualExtendTableModelVerbalRuleGeometric2a.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsHard/tSolveGeometricProblems3a.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsHard/tSolveGeometricProblems3b.1
ecd://K12Math/Formative/visualExtendVerbalRuleModelGeometricTask/visualExtendVerbalRuleModelGeometricMedium/tVisualExtendVerbalRuleModelGeometric2a.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricMedium/tTableExtendGeometric2a.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricMedium/tVisualExtendGeometric2a.1
ecd://K12Math/Formative/verbalRuleExtendModelGeometricTask/verbalRuleExtendModelGeometricEasy/tVerbalRuleExtendModelGeometric1a.1
ecd://K12Math/Formative/verbalRuleExtendModelGeometricTask/verbalRuleExtendModelGeometricMedium/tVerbalRuleExtendModelGeometric2a.1
ecd://K12Math/Formative/verbalRuleExtendModelGeometricTask/verbalRuleExtendModelGeometricEasy/tVerbalRuleExtendModelGeometric1b.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricMedium/tTableExtendGeometric2b.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricHard/tTableExtendGeometric3a.1
ecd://K12Math/Formative/commonRatioTask/commonRatioMedium/tCommonRatio2a.1
ecd://K12Math/Formative/commonRatioTask/commonRatioHard/tCommonRatio3a.1
ecd://K12Math/Formative/commonRatioTask/commonRatioHard/tCommonRatio3b.1
ecd://K12Math/Formative/commonRatioTask/commonRatioEasy/tCommonRatio1a.1
ecd://K12Math/Formative/commonRatioTask/commonRatioEasy/tCommonRatio1b.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricHard/tTableExtendGeometric3b.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricHard/tVisualExtendGeometric3a.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricMedium/tModelExtendTableGeometric2a.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricEasy/tVisualExtendGeometric1a.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricMedium/tExtendGeometric2c.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricMedium/tExtendGeometric2a.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricHard/tExtendGeometric3a.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricEasy/tTableExtendGeometric1a.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricHard/tExtendGeometric3b.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricEasy/tModelExtendTableGeometric1a.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricEasy/tVisualExtendGeometric1b.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricMedium/tExamplesGeometric2a.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricEasy/tTableExtendGeometric1b.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricEasy/tVisualExtendGeometric1c.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricEasy/tModelExtendTableGeometric1b.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricEasy/tExtendGeometric1a.1
ecd://K12Math/Formative/verbalRuleModelGeometricTask/verbalRuleModelGeometricHard/tVerbalRuleModelGeometric3a.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricMedium/tExamplesGeometric2b.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricEasy/tExtendGeometric1c.1
ecd://K12Math/Formative/verbalRuleExplicitModelGeometricTask/verbalRuleExplicitModelGeometricMedium/tVerbalRuleExplicitModelGeometric2a.1
ecd://K12Math/Formative/verbalRuleExplicitModelGeometricTask/verbalRuleExplicitModelGeometricHard/tVerbalRuleExplicitModelGeometric3a.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricMedium/tModelExtendTableGeometric2b.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricHard/tExamplesGeometric3a.1
ecd://K12Math/Formative/visualExplicitVerbalRuleModelGeometricTask/visualExplicitVerbalRuleModelGeometricHard/tVisualExplicitVerbalRuleModelGeometric3a.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricHard/tModelExtendTableGeometric3a.1
ecd://K12Math/Formative/visualExplicitVerbalRuleModelGeometricTask/visualExplicitVerbalRuleModelGeometricHard/tVisualExplicitVerbalRuleModelGeometric3b.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricHard/tModelExtendTableGeometric3b.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricHard/tExamplesGeometric3b.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricEasy/tExamplesGeometric1a.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricMedium/tRecursiveRuleGeometric2a.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricEasy/tExamplesGeometric1b.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricMedium/tRecursiveRuleGeometric2b.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricHard/tRecursiveRuleGeometric3a.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricHard/tRecursiveRuleGeometric3b.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricEasy/tRecursiveRuleGeometric1a.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricEasy/tRecursiveRuleGeometric1b.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricMedium/tExplicitGeometric2a.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricMedium/tExplicitGeometric2b.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricHard/tExplicitGeometric3a.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricHard/tExplicitGeometric3b.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricEasy/tExplicitGeometric1a.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricEasy/tExplicitGeometric1b.1ACED development and data collection was sponsored by National Science Foundation Grant No. 0313202.