Tutorial: Bayesian Networks in Educational Assessment

Russell Almond, ETS
Robert Mislevy, University of Maryland
David Williamson, ETS
Duanli Yan, ETS

This is a collection of material related to our 2008 NCME Tutorial. This will also be available via CD-ROM and memory stick at the tutorial. See you in New York!

Instructions for Attendees. There is now a "live computing" exercise included in the seminar. To do this we are recommending everybody who can bring a laptop capable of running some version of Windows. (If you don't have a laptop, hopefully you will be able to share with somebody who does. A Mac laptop capable of running classic will also work.) We are also recommending you do the following two steps:

1. Download student/demonstration version of the software Netica from Norsys. (Other possible software packages are listed below, but we will be preparing the exercises in Netica.)
2. Download the example networks to be used. All Session III networks (Netica) and All Session V networks (Netica)

• Abstract
• Slides and Handouts
• Links to Software
• Links to other resources

Abstract

As consumers of assessments become more sophisticated, they start demanding more from their assessments. In particular, they want more in the way of diagnostic information and more sophisticated constructed response tasks. Both of these require multidimensional measurement models.

Bayesian networks are a technique for managing multidimensional models. By representing the variables of the model as nodes in the graph and using edges in the graph to represent patterns of dependence and independence among the variables, the model graph serves as a bridge between educational and psychometric experts, and further helps the computer derive efficient computational strategies.

This tutorial first reviews the literature on Bayesian networks from the statistics and artificial intelligence communities in which they are quite popular. Then it looks at specific applications of these models to educational testing. In particular, it describes the relationship between these models and ETS's Evidence Centered Design methodology.

Slides and Handouts

I. Evidence Centered Design

Covers definitions of Proficiency, Evidence and Task models, example of model construction and four process model. Slides (PDF) Handout (PDF).

II. Graphical Models

Basic graph theory definitions, comparisons with related models, and the main propagation algorithm. Slides (PDF) Handout (PDF).

III. Graphical Modeling Tools and Applications

Includes simple examples in Netica and software survey Slides (PDF) Handout (PDF). Additionally the following data is used in the hands on exercise.

• All Session III networks (Netica)
• The IRT 5 example is available in a number of other formats.
• 3 Variable IRT model (Smile)
• 5 Variable IRT model (Ergo)
• 5 Variable IRT model (Hugin)
• 5 Variable IRT model (MSBNx)
• 5 Variable IRT model with Context Variable (MSBNx)
• Combination Type Example (MSBNx). Simple example of compensatory, conjunctive and disjunctive models.

IV. Refining Graphical Models with Data

Includes parameterization, MCMC overview, Model Criticism and cautionary note on learning causality. Slides (PDF) Handout (PDF).

V. ACED: ECD in Action

Describes an NSF sponsored assessment for learning system called ACED which uses Bayes net scoring. Includes exercises based on the ACED scoring model and data. Slides (PDF) Handout (PDF). All Session V networks (Netica)

Bibliography

Bayes net and ECD Bibliography

The handout version is also available as one big file containing all sessions and the bibliography. Honkin' big handout (PDF).

On-line Resources

For quick reference, here are the on-line resources referenced in the bibliography.

Computer programs and documentation available on the Web:

This is a partial list of software packages we have used or think are worth paying attention to. A more complete list is available in Kevin Murphy's survey of Bayes net software at: http://www.cs.ubc.ca/~murphyk/Bayes/bnsoft.html

MSBNx (Microsoft Bayesian Network editor).

http://research.microsoft.com/adapt/MSBNx/default.asp MSBNx is free for research and educational purposes. Commercial applications are not supported.

HUGIN Expert System site.

http://www.hugin.com/HUGIN is a computational engine for discrete Bayes nets, also has many learning tools. This site has tutorials, examples, and a demo version of HUGIN that can be downloaded and used to run the examples. See the "Developer" pages for these things.

Netica (Norsys Software Crop)

http://www.norsys.com/ Netica is another very complete commercial grade Bayes net engine, includes some learning tools.

Genie/Smile (Decision Systems Lab, Univ. of Pittsburgh)

http://genie.sis.pitt.edu/ Open source project, free under Gnu Public License. Also contains a ``translator'' which translates between network formats.

ERGO (Noetic Systems)

http://www.noeticsystems.com/ Downloadable demo version of the ERGO program, for constructing and reasoning with Bayes nets. We have used Ergo successfully in the past for several projects, but no new version has been released lately.

PNL: Probabilistic Networks Library (Intel)

https://sourceforge.net/projects/openpnl/ A work in progress open source library of Bayes net engine software. Available under a Berkeley license.

Useful (Bayesian) Statistical Software

BUGS (Bayesian inference Using Gibbs Sampling).

http://www.mrc-bsu.cam.ac.uk/bugs/welcome.shtml Downloadable version for Windows. See also OpenBUGS http://mathstat.helsinki.fi/openbugs/

JAGS (Just Another Gibbs Sampler)

https://sourceforge.net/projects/mcmc-jags/ A rewrite of Classic BUGS (command line only, no GUI support) that runs under Linux, MacOS X, and Windows.

FBM: Flexible Bayesian Modeling

http://www.cs.utoronto.ca/~radford/fbm.software.html Radford Neal's Flexible Bayesian Modeling and Markov Chain Sampler.

R

http://www.r-project.org/ General purpose statistical computing environment based on S language.

Other On-Line Resources:

ECD Wiki

http://ecd.ralmond.net/ecdwiki/ Email Russell to get a password to contribute to the discussion.

Heckerman tutorial on learning (Heckerman, D. [1995])

ftp://ftp.research.microsoft.com/pub/tr/tr-95-06.pdf Note: Other Microsoft Research technical reports are available on-line from http://www.research.microsoft.com/

Association for Uncertainty in Artificial Intelligence home page

http://www.auai.org/ UAI conference proceedings is the most important publication in this area.

CRESST Technical Report Archive

http://www.cse.ucla.edu/products/reports.asp Early versions of many of the Mislevy references (including in press references) are available here. (Hint: search for ``Mislevy''). The CRESST web site changes frequently, so this link may be out of date. If the link is broken, google "CRESST Reports".

Bob Mislevy's Home page

http://www.education.umd.edu/EDMS/mislevy/ Course handouts and paper pre-prints.

CiteSeer Cross-Reference Database

http://citeseer.ist.psu.edu/cis On-line cross reference database with lots of articles on Bayes nets. Many of the bibliography entries are available through CiteSeer.

Copyright

All handouts and slides from this tutorial are unpublished work Copyright 2002--8 by Educational Testing Service.

These materials are an unpublished, proprietary work of ETS. Any limited distribution shall not constitute publication. This work may not be reproduced or distributed to third parties without ETS's prior written consent. Submit all requests through http://www.ets.org/legal/copyright.html.

ACED development and data collection was sponsored by National Science Foundation Grant No. 0313202. Thanks to Val Shute for permission to use ACED data in this tutorial.