Value of Testing
This example is an adaptation of a classic Influence Diagram? model called the Oil Wildcatter's problem. It specifically illustrates the concept of value of information. The adaptation puts the example in an educational context.
The network
Start by looking across the bottom of the figure. There are two nodes, "Skill and Time 1" and the "Skill at Time 2", both of which have two states High
and Low
. There is some chance of improvement over time, indicated by the arrow, and presumably higher skills are better. The pink utility node, "Benefit of final result" indicates how much better it is to have a student at the higher skill level. (Without loss of generality, we can set the utility of the student being in the low state to zero.)
In the middle layer, there is a blue decision node "Instruction", with two states Tier1
and Tier2
. Tier2
is a more intense instruction which should have a higher probability of moving a student from Low
to High
on the skill. This is represented by the arrow from "Instruction" to "Skill at Time 2". Tier2
presumably also costs more. This is reflected in the pink utility node "Cost Of Instruction". Once again, without loss of generality, we can set the cost of Tier1
to zero.
As Tier2
instruction costs more, presumably we only want to give it to students who are Low
at time 1, but we cannot directly observe the variable "Skill at Time 1". Instead, we have the option of giving a "Screening Test". Note that this has three values as it could be Unknown
. Note that it is also a parent of "Instruction" as presumably the instructional decision will be made with the test result in hand (if the test was given). There is also a decision about whether or not to test (the decision node "Test?"). The test also has a cost, hence the pink utility node "CostOfTesting". Once again, we set the cost of not testing to zero.
The utilities
Utilities represent costs and benefits in an InfluenceDiagram?. By convention they are drawn as hexagons, and by default Netica? colors them pink. Costs are negative utilities, and hence appear the same way in the diagram.
The total utility is the sum of all the utility nodes. Note that this rather assumes that all of the utilities are on the same scale. Putting all of the costs and utilities on the same scale is often quite difficult. In the example, the cost of testing is related to both money payed for the use of the test, and seat time of the student taken away from instruction. The cost of the Tier2
instruction involves both student and instruction time as well as possible additional expendable learning materials. The benefit of the student being in the higher skill level is quite nebulous.
The decision nodes
The blue decision nodes behave somewhat differently from the yellow chance nodes. (In a traditional influence diagram, chance nodes are round and decision nodes are square, Netica? uses color instead of shape.) In particular, it is assumed that the decision maker will pick the value for the decision node the maximizes the expected utility. Indeed the solution to an influence diagram, is a "policy" a series of rules for assigning values to the decision variables based on previously seen values.
The arrows coming into a decision node indicate information that is available at the time the decision will be made. Thus the "Instruction" node has the "Screening Test" node as a parent.
The values show in the "Test?" node are the expected utilities when you test and when you do not test, respectively. If you select a value for "Test?", Netica will then display the expected utilities for Tier1
and Tier2
instruction.
Value of Information
If we set the cost of testing to zero (as is done in the picture above) the difference between the utilities Yes
and No
is the "value of information". This is the maximum we should be willing to pay for the test.
If we add a link directly from "Skill at Time 1" to "Instruction", then the difference between the utility for Tier1
and Tier2
will give us the "value of perfect information". This is the most we should be willing to pay for "any" test.
The network
The parameters
The two networks shown below differ from the first because the conditional probability tables and utility functions have parameters. The yellow nodes seemingly unattached to the main network are parameters. In Netica? you can use them to explore the sensitivity of the decision to the parameters. To do this enter a new value for the parameter (using "Enter Numeric findings" on the right click menu) and then click the "equation to table" button (looks like a tree with f() above it) and then the lighting bolt to recompile the network.
Version 1, with parameters for the utilities only
This version has parameters for the utilities, but not the conditional probability tables.
Netica Network
- The pink utility node, "Benefit of final result" indicates how much better and the yellow parameter node "MarginalBenefit of High Skill" indicates how much better it is to have a student at the higher skill level.
- The pink utility node "Cost Of Instruction" indicates the relative cost of instruction. Once again, without loss of generality, we can set the cost of
Tier1
to zero, so this utility has a single parameter "Marginal Cost of Tier 2".
- The test also has a cost, hence the pink utility node "CostOfTesting". Once again, we set the cost of not testing to zero, and this utility has a single parameter, "Cost of Testing".
Version 2, with parameters for the utilities and probabilities
This version has both parameters for the conditional probability tables and the utilties.
Netica Network
- The probability that "Skill at Time 1" is
High
is indicated by the "Base Rate" parameter.
- The effect of the instruction is given in the conditional probability of "Skill at Time 2" given "Skill at Time 1" and "Instruction". This diagram uses a no forgetting model (so the if the student is
High
at Time 1, they stay High
at Time 2. Therefore, the conditional proability table has two parameters: "Tier 1 effect" and "Tier 2 effect", which are the probabilities that a student who is Low
at Time 1 will move to High
given that "Instruction" was Tier1
or Tier2
respectively. (Presumably "Tier 2 effect" is higher than "Tier 1 effect", or the lower cost Tier1
would always be better.
- The properities of the test are given by two numbers, the "Sensitivity" (the probability of a
Pass
result given that the student is High
) and the "Specificity" (the probability of a Fail
result given that the student is Low
). (For more information about sensitivity and specificity, see https://en.wikipedia.org/wiki/Sensitivity_and_specificity.)