Risk Assessment and Key Indicator Matrices Decision Theory and Revised Algorithms to deal with False Negatives

Risk Assessment and Key Indicator methodologies are two approaches utilized in differential monitoring systems for generating an abbreviated inspection by only looking at a core set of rules based upon statistical predictor or risk assessment algorithms. In this post the matrices (pictured below) utilized to generate these core sets of rules are depicted and with a matrix that determines their respective validation status based upon subsequent studies.

The first matrix (KIM Matrix) deals with the Key Indicator Methodology (KIM) and demonstrates how key indicator rules are determined by measuring each potential rule and comparing it to the regulatory compliance history for the respective set of all rules for a given jurisdiction in which the programs are grouped into either a high (Full or substantial regulatory compliance with all rules) or low compliant groups (several or more violations of rules). From the matrix, it is clear that for a rule to become an indicator rule, there needs to be a very high correlation between the rule being in compliance with the high group and out of compliance with the low group. It is only when this occurs that the rule will distinguish between high and low compliance and be a predictor rule. The other two cells should occur less frequently but there will be some occurrences when these do occur and when they do, these rules will not make the threshold of becoming indicator rules. So Key Indicator Predictor Rules increase performance by predicting overall regulatory compliance.

The second matrix (RAM Matrix) deals with the Risk Assessment Methodology (RAM) and demonstrates how risk assessment rules are determined by measuring each potential rule by the amount of risk of morbidity or mortality a client is placed in because of non-compliance with the specific rule and how likely will this occur. As one can see, the cell which contains high risk rules and they are likely to occur would be included on the risk assessment tool. All the other cells are color coded in decreasing risk and likelihood categories and a jurisdiction can determine the appropriate thresholds. More risk rules would be included for a risk aversive approach while less risk rules would be included for a more lenient approach or because the number of key indicator rules are sufficient to insure the health and safety of the clients being served. So Risk Assessment Rules decrease risk to clients but are not predictive rules of overall regulatory compliance.

The last matrix (KIM/RAM Validation Matrix) is used after the KIM and RAM tools are actually used to validate that they are working as intended. KIM should be statistically predicting overall compliance with all the rules (Rules in Compliance cell), while RAM should be mitigating risk in the program by always having the high risk rules in compliance (also Rules in Compliance cell). Part of the KIM validation strategy is that the opposite should also be occurring in that when the KIM tool has indicator rules out of compliance, it should statistically predict rules out of compliance with other rules (the Rules Out of Compliance cell). Something that can occur but needs to be eliminated are the false negatives in which the KIM is in compliance but there is non-compliance detected elsewhere in the rules. When full compliance is used for the high compliant group in the KIM Matrix, this eliminates this from happening. But if substantial compliance is used as the criterion for the high compliant group, then this can become problematic. If substantial compliance is used as the threshold for the high compliant group, a multiplier needs to be applied to rule out the likelihood of false negatives (please see the blog post on this algorithm adjustment posted back in January of this year or look at the description provided below the matrices). False positives are possible also, but are not of overall concern from a safety point of view but are a concern from a psychometric standpoint and additional research needs to be done to determine the cause.

Hopefully this post helps licensing administrators, licensing researchers, and regulatory scientists to see the logic behind the differential monitoring methodologies of key indicator and risk assessment and how best to take advantage of both.