Recent Changes - Search:
 HomePage

ECD

ACED

Bayesian Networks in Educational Assessment

Cognition and Assessment

PmWiki

pmwiki.org

edit SideBar

QMatrix

ACED Evidence Models, Assembly Model and Q-Matrix

Evidence Models

All of the ACED tasks have a single observable outcome variable that is scored dichotomously (correct or incorrect). For this reason each Evidence Model? corresponds to a unique row pattern in the Q-Matrix?. These are replicated according to the number of tasks that use that evidence model for scoring (i.e., the number of tasks built from the Task Model? corresponding to the Evidence Model).

Evidence Model Level Q-Matrix
Task/Evidence ModelSolve Geometric ProblemsCommon RatioExampleExplicitExtendModel!RecursiveTableVerbalVisualDesign PatternAnchor
Solve Geometric Problems++000000000CompensatorySGP
Common Ratio0++00000000CompensatoryCommon Ratio
Examples Geometric00++0000000CompensatoryExamples
Explicit Geometric000++000000CompensatoryExamples
Extend Geometric0000++00000CompensatoryExtend
Model Extend Table Geometric0000+++0+00CompensatoryModel
Recursive Rule Geometric000000++000CompensatoryRecursive
Table Extend Geometric0000+00++00CompensatoryTable
Verbal Rule Explicit Model Geometric000+0+00++0CompensatoryVerbal
Verbal Rule Extend Model Geometric0000++00++0CompensatoryVerbal
Verbal Rule Model Geometric00000+00++0CompensatoryVerbal
Visual Explicit Verbal Rule Model Geometric000+0+00+++CompensatoryVisual
Visual Extend Geometric0000+0000++CompensatoryVisual
Visual Extend Table Model Verbal RuleGeometric0000++0++++CompensatoryVisual
Visual Extend Verbal Rule Model Geometric0000++00+++CompensatoryVisual

The + indicates that the corresponding proficiency variable (names are abbreviated) is relevant, the ++ indicates that the variable is the primary one for this task type.

The names of the task and evidence models are created by removing the spaces from the names, making the first letter lower case, and appending the work 'Task' (for task models) or 'Evidence' (for evidence models).

The design pattern of 'Compensatory' means that the experts thought the skills entered in a compensatory manner.

The anchor column indicates which Scale Anchor? set this evidence model type belongs to.

The Solve Geometric Problems tasks are actually tasks about distinguishing geometric from other types of sequences, so the proficiency variable is not correctly names.

Difficulty Targets and Link Model Priors

The ACED team decided to build three different variants of of each type of task: an easy, a medium difficulty and a hard task. To model this, we used a variant of the evidence model called the Link Model Prior?, a sub-model that differs from its evidence model parent by having a different prior distribution for the parameters. In this case "easy" tasks were thought to have a difficulty of around -1 (on an IRT scale); "hard" tasks were thought to have a difficulty of around 1; and "medium" tasks were thought to have a difficulty of around 0.

Notationally, the Link Model Prior names were formed from the Evidence Model names by adding 'Hard', 'Medium', or 'Easy', and 'LMP'. The Task Specifications? (a sub-model of the task model) names were formed by adding just the name. The task names ended in a number indicating the difficulty target (1=easy, 2=medium, 3=hard).

The "Solve Geometric Problems" tasks

In the initial design of ACED, the the team identified one class of problems as "Solve Geometric Problems" tasks and their Q-Matrix entry indicated that they tapped the "Solve Geometric Problems" proficiency model (the highest level one). In later analysis, these tasks proved to be highly influential. On closer scrutiny, it turns out that the tasks involved recognizing whether or not a sequence was geometric. Thus, maybe the proficiency model was really missing a "Recognize Sequence Type" node, which is the one this should be attached to. The original model was the one used for task selection and analysis in all of the papers.

(Note that the CBAL Mathematics Team at ETS, noted the prevalence of a pattern in which students would try to solve proportional reasoning problems using addition instead of multiplication. It may be the sequence which typically forced the recognize sequence problems first, helped students later. Thoughts?)

The complete task list

ACED used an extended Task ID system (designed for StatShop) that used a URL-like syntax to indicate the ECD models associated with a task. The extended IDs look like:

ecd://"Project"/"Assessment"/"Model"/"Submodel"/"ID"."Version"

In this case "Project" is always K12Math, "Assessment" is always Formative and "Version" is always 1. The "ID" part is unique, always starts with a 't' and ends with a number indicating difficulty. This shorter ID is used in a number of places.

The complete list of tasks is as follows:

ecd://K12Math/Formative/commonRatioTask/commonRatioMedium/tCommonRatio2b.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsMedium/tSolveGeometricProblems2a.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsMedium/tSolveGeometricProblems2b.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsEasy/tSolveGeometricProblems1a.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsEasy/tSolveGeometricProblems1b.1
ecd://K12Math/Formative/visualExtendTableModelVerbalRuleGeometricTask/visualExtendTableModelVerbalRuleGeometricMedium/tVisualExtendTableModelVerbalRuleGeometric2a.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsHard/tSolveGeometricProblems3a.1
ecd://K12Math/Formative/solveGeometricProblemsTask/solveGeometricProblemsHard/tSolveGeometricProblems3b.1
ecd://K12Math/Formative/visualExtendVerbalRuleModelGeometricTask/visualExtendVerbalRuleModelGeometricMedium/tVisualExtendVerbalRuleModelGeometric2a.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricMedium/tTableExtendGeometric2a.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricMedium/tVisualExtendGeometric2a.1
ecd://K12Math/Formative/verbalRuleExtendModelGeometricTask/verbalRuleExtendModelGeometricEasy/tVerbalRuleExtendModelGeometric1a.1
ecd://K12Math/Formative/verbalRuleExtendModelGeometricTask/verbalRuleExtendModelGeometricMedium/tVerbalRuleExtendModelGeometric2a.1
ecd://K12Math/Formative/verbalRuleExtendModelGeometricTask/verbalRuleExtendModelGeometricEasy/tVerbalRuleExtendModelGeometric1b.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricMedium/tTableExtendGeometric2b.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricHard/tTableExtendGeometric3a.1
ecd://K12Math/Formative/commonRatioTask/commonRatioMedium/tCommonRatio2a.1
ecd://K12Math/Formative/commonRatioTask/commonRatioHard/tCommonRatio3a.1
ecd://K12Math/Formative/commonRatioTask/commonRatioHard/tCommonRatio3b.1
ecd://K12Math/Formative/commonRatioTask/commonRatioEasy/tCommonRatio1a.1
ecd://K12Math/Formative/commonRatioTask/commonRatioEasy/tCommonRatio1b.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricHard/tTableExtendGeometric3b.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricHard/tVisualExtendGeometric3a.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricMedium/tModelExtendTableGeometric2a.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricEasy/tVisualExtendGeometric1a.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricMedium/tExtendGeometric2c.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricMedium/tExtendGeometric2a.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricHard/tExtendGeometric3a.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricEasy/tTableExtendGeometric1a.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricHard/tExtendGeometric3b.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricEasy/tModelExtendTableGeometric1a.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricEasy/tVisualExtendGeometric1b.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricMedium/tExamplesGeometric2a.1
ecd://K12Math/Formative/tableExtendGeometricTask/tableExtendGeometricEasy/tTableExtendGeometric1b.1
ecd://K12Math/Formative/visualExtendGeometricTask/visualExtendGeometricEasy/tVisualExtendGeometric1c.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricEasy/tModelExtendTableGeometric1b.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricEasy/tExtendGeometric1a.1
ecd://K12Math/Formative/verbalRuleModelGeometricTask/verbalRuleModelGeometricHard/tVerbalRuleModelGeometric3a.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricMedium/tExamplesGeometric2b.1
ecd://K12Math/Formative/extendGeometricTask/extendGeometricEasy/tExtendGeometric1c.1
ecd://K12Math/Formative/verbalRuleExplicitModelGeometricTask/verbalRuleExplicitModelGeometricMedium/tVerbalRuleExplicitModelGeometric2a.1
ecd://K12Math/Formative/verbalRuleExplicitModelGeometricTask/verbalRuleExplicitModelGeometricHard/tVerbalRuleExplicitModelGeometric3a.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricMedium/tModelExtendTableGeometric2b.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricHard/tExamplesGeometric3a.1
ecd://K12Math/Formative/visualExplicitVerbalRuleModelGeometricTask/visualExplicitVerbalRuleModelGeometricHard/tVisualExplicitVerbalRuleModelGeometric3a.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricHard/tModelExtendTableGeometric3a.1
ecd://K12Math/Formative/visualExplicitVerbalRuleModelGeometricTask/visualExplicitVerbalRuleModelGeometricHard/tVisualExplicitVerbalRuleModelGeometric3b.1
ecd://K12Math/Formative/modelExtendTableGeometricTask/modelExtendTableGeometricHard/tModelExtendTableGeometric3b.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricHard/tExamplesGeometric3b.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricEasy/tExamplesGeometric1a.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricMedium/tRecursiveRuleGeometric2a.1
ecd://K12Math/Formative/examplesGeometricTask/examplesGeometricEasy/tExamplesGeometric1b.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricMedium/tRecursiveRuleGeometric2b.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricHard/tRecursiveRuleGeometric3a.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricHard/tRecursiveRuleGeometric3b.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricEasy/tRecursiveRuleGeometric1a.1
ecd://K12Math/Formative/recursiveRuleGeometricTask/recursiveRuleGeometricEasy/tRecursiveRuleGeometric1b.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricMedium/tExplicitGeometric2a.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricMedium/tExplicitGeometric2b.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricHard/tExplicitGeometric3a.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricHard/tExplicitGeometric3b.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricEasy/tExplicitGeometric1a.1
ecd://K12Math/Formative/explicitGeometricTask/explicitGeometricEasy/tExplicitGeometric1b.1


HomePage ACED Data Model People Publications

ACED development and data collection was sponsored by National Science Foundation Grant No. 0313202.

Edit - History - Print - Recent Changes - Search
Page last modified on July 06, 2012, at 01:01 PM