...
Weekly Coding Calls: Assessor staff members are required to participate in weekly coding calls to discuss any scoring questions that the assessor Assessor may have unless there is a scheduled assessment occurring during the scheduled time. Coding calls help assessors Assessors maintain reliability in scoring measures and prevent drift in scoring.
Field-Based Inter-Rater Reliability: As part of the quality assurance strategy, a field-based inter-rated reliability routine focusing on star-level ratings will be implemented each quarter. The field-based inter-rater reliability routine is designed to supplement the existing ATCP reliability routines, which focus on individual classroom observations and document scoring rather than a full, field-based site assessment. In addition to the quarterly reliabilities which occur through the ATCP, this includes double assessment of 2% of programs annually (approximately 120 programs) for the field-based inter-rater reliability routine. These programs will be scored by independent raters at the same time to ensure the overall star ratings have stability from rater to rater. The 120 ELPs will be randomly selected for full site assessments (all full site assessments can be considered as part of the field-based inter-rater reliability) using proportional stratified sampling across small, medium, and large schools. Raters deployed will include the routine assigned assessor Assessor (rater 1) and an assessment manager, senior manager, or lead assessor Lead Assessor (rater 2). Assessors unable to maintain reliability will be removed from the pool of gold standard raters – see additional guidance included in the quarterly reliability test. Results of the field-based reliability monitoring routine will inform CAE and TWC about the overall consistency of star ratings, building confidence in the sensitivity and accuracy of the Texas Rising Star assessment instrument and procedures. Should areas of weakness be identified, the team will work to identify the root cause and provide recommendations for improvement. An analysis of the data collected will be conducted by the Fidelity monitoring team. A quarterly quality assurance report will be created within thirty calendar days of the end of each quarter.