...
For Assessor Managers to better support their Assessor staff and to conduct validity and reliability practices, Managers are also required to attain certification and maintain reliability via quarterly exercises.
Quarterly Reliability Coding Practices
Assessors are given three opportunities to demonstrate continued reliability in scoring through quarterly reliability (QR) coding. QR has two coding modules. One module consists of facility assessment (Categories 1 and 3) for a home or center. The second module will be a classroom observation (Categories 2 and 4) for one of the five age groups (infant, toddler, preschool, school-age, or all ages-home). Each assessor is contacted via email prior to their QR coding date informing them that QR timeframe is forthcoming and their allotted coding period (typically 30 calendar days). The assessor should look at their schedule to determine when they will devote time to complete their assigned QR modules, allowing for the potential of PLCs and retesting to occur as needed and communicate with the managers. The intent is for Assessors to complete the initial assigned modules within two weeks of receiving the module assignment. Subsequent modules (if any are assigned) should be completed within the following week. Each Assessor will receive additional guidance from their managers to support completion of retests if needed. Assessors will track the completion of each module and PLCs. A report will be available via the Power-BI that will include quarterly reliability updates for each participating Assessor. Certification and quarterly reliability coding exercises are independent practices and should not be completed with other team members. Assessors should also refrain from sharing scores with team members as well.
Those assessors who are unable to pass one or both of their QR testing modules (after 3 attempts) will be notified via email by TWC, along with their managers, regarding next steps. If the assessor fails one or both parts of the testing (facility or classroom), then they will be asked to shadow a seasoned assessor who is maintaining reliability, for those categories of any upcoming assessments or annual monitoring visits (thus not able to complete assessments on those categories on their own), until they are able to show reliability by completing the next reliability test available to them, with a passing result. If an Assessor does not pass the three attempts, they will be placed on a Performance Improvement Plan for not passing the requirements of the reliability test. After a specified period of shadowing, the Assessor will be assigned to participate in the next group’s QR testing and given another opportunity to demonstrate reliability. During the shadowing period, the assessor will be assigned to shadow several experienced assessors a minimum of two times per week. The remainder of the week, the assessor will focus on completing mock data entry and comparing their scores to the assessor they were assigned to shadow. Assessors will also be assigned supplemental learning content to review to support knowledge in the areas needed. Managers will check-in with staff who are on a performance plan at least once a week.
It is important that all assigned quarterly reliability modules are completed during the assigned period to ensure timely analysis of data and to prevent any disruption in the Assessor completing any scheduled assessments.