Regulatory Compliance Scale Trials and Tribulations

There have been several posts in these RIKNotes dealing with the Regulatory Compliance Scale (RCS) that was proposed as an alternative measurement strategy to regulatory compliance violation frequency data where the number of rule/regulation violations are counted. The RCS has recently been tested in validation studies to determine the proper thresholds for its scaling. The attached report provides the methodology used and the results of these validation studies conducted in the USA and Canada. A couple of footnotes which I noticed after my initial posting: The Fibonacci Sequencing is a modification of the original, I took the liberty to deal with the extremes of the sequence in order to increase the variance in the scaling which is a predominant problem with regulatory compliance data. The second footnote is that the RCS based upon empirical data as well as anecdotal reports is on an equal par with regulatory compliance violation frequency count data (RCV) but the only way to test the RCS is for licensing agencies to set up these types of analyses comparing them side by side and then determining which is better to use. I am not suggesting that the RCS be used in place of RCV. The data from this report just does not support doing that. I would also ask licensing agencies to send me the results of their studies so that I can add those data to the ever expanding international database being maintained at the Research Institute for Key Indicators Data Laboratory at Penn State University. Please just send any empirical results to rfiene@rikinstitute.com.

The Regulatory Compliance Scale (RCS) was introduced several years ago and has been used in a couple of validation studies for differential monitoring and regulatory compliance’s ceiling effect phenomenon.  RCS buckets or thresholds were statistically generated based upon these studies, but it is time to validate those buckets and thresholds to determine if they are really the best model in creating a regulatory compliance scale.  Since proposing the RCS, there has been a great deal of interest from jurisdictions in particular from Asian and African nations.  Additional statistically based trials were conducted, and this brief report is the compilation of those trials over the past year. 

The data used are from several jurisdictions that are part of the international database maintained at the Research Institute for Key Indicators Data Laboratory at Penn State University focusing on program quality scores and rule violation frequency data.  These data from the respective databases were recoded into various thresholds to determine the best model.  The jurisdictions were all licensing agencies in the US and Canada geographically dispersed where both regulatory compliance and program quality data was obtained from a sample of early care and education programs. 

METHODOLOGY 

The following methodology was used starting with the original RCS buckets/thresholds of Full, Substantial, Medium, and Low regulatory compliance: 

Table 1: RCS Models used for analyses 

RCS				Models
		Original	1	2	3	4	5
	Full	100	100	100	100	100	100
Scaling	Substantial	99-98	99-97	99-97	99-98	99-98	99-97
	Medium	97-90	96-90	96-93	97-95	97-85	96-85
	Low	89>	89>	92>	94>	84>	84>

Five alternate models were used to compare the results to the original RCS.  The numbers indicate the number of violations subtract from a perfect score of 100.  Full regulatory compliance indicates no violations and a score of 100 on the scale.  The next bucket of 99-98 indicates that there were 1 or 2 regulatory compliance violations which resulted in a 99-98 score on the scale.  This logic continues with each of the models.   

The scale score was determined in the following manner: Full Regulatory Compliance = 7; Substantial Regulatory Compliance = 5; Medium Regulatory Compliance = 3; and Low Regulatory Compliance =1.  This rubric is how the original RCS scaling was done on a Likert type scale similar to other ECE program quality scales, such as the Environmental Rating Scales. 

RESULTS 

The following results are correlations amongst the respective RCS Models from Table 1 compared to the respective jurisdictions program quality tool (Quality1-3): ERS or CLASS Tools. 

Table 2: RCS Model Results compared to Quality Scales 

RCS results	Models	Quality1	Quality2	Quality3
Jurisdiction1	RCS0	.26*	.39*	.39*
	RCS3	.21	.32*	.33*
	RCS5	.20	.36*	.33*
Jurisdiction2	RCS0	.76**	.46**	—
	RCS3	.12	-.07	—
	RCS5	.18	-.02	—
	RCSF1	.55**	.29*	—
	RCSF2	.63**	.34	—
Jurisdiction3	RCS0	.19	.18	.16
	RCS3	.21	.21	.15
	RCS5	.18	.16	.07
	RCSF1	.17	.17	.10
	RCSF2	.18	.18	.19
Jurisdiction4	RCS0	.24*	—	—
	RCS3	.28*	—	—
	RCS5	.30*	—	—
	RCSF1	.21	—	—
	RCSF2	.29*	—	—
Jurisdiction5	RCS0	.06	-.02	.07
	RCS3	.06	-.01	.05
	RCS5	.08	.00	.09
	RCSF1	.00	-.03	.05
	RCSF2	.05	-.03	.05

*Statistically significant .05 level;  

**Statistically significant .01 level. 

In the above table starting under Jurisdiction2, two new models were introduced based upon the Fibonacci Sequence (Fibonacci1 = RCSF1; Fibonacci2 = RCSF2) and their model structure is in the following Table 3.  The reason for doing this is that the Fibonacci Sequence introduces additional variation into the scaling process. 

Table 3: RCS Fibonacci Models 

RCS Fibonacci			Models
		Original	Fibonacci1	Fibonacci2
	Full	100	100	100
Scaling	Substantial	99-98	40	90
	Medium	97-90	20	20
	Low	89>	13	13

DISCUSSION 

Based upon the above results, it appears that the original RCS model proposed in 2021 is still the best model to be used, although the Fibonacci Sequence model is a close second in some of the jurisdictions.  This model will need further exploration in determining its efficacy as a replacement or enhancement to the original RCS Model. 

The bottom line is that the original RCS Model is as good as any and no other model is consistently better than all the rest.  The RCS Model does have a slight edge over Regulatory Compliance Violation RCV frequency counts.  So, the recommendation would be for licensing agencies to think in terms of using this new scaling technique in one of its model formats. 

I have updated the attached paper with appendices dealing with data distributions and basic descriptives that licensing researchers and regulatory scientists may find interesting and appealing in considering this particular approach.