Improving Teacher Shortage Metrics: A Comprehensive Approach
News reports often paint a grim picture of the teacher labor market, labeling it as “dire” and “in crisis.” The demand for qualified teachers far exceeds the supply, leading to what many describe as “unprecedented hardships” and “critical shortages” in schools across the country. However, a closer look at the data reveals a more nuanced reality.
Teacher shortages are not uniform, and the challenges vary significantly by subject area and school type. While the public perception of teacher shortages tends to lack these nuances, policymakers often lack detailed information to address the specific needs of schools. This gap in understanding highlights the importance of gathering accurate and timely data on teacher demand.
One low-cost solution to obtaining a real-time snapshot of the teacher labor market is web scraping. This automated data-extraction technique allows for the regular export and refresh of data from the internet, creating updated datasets for analysis and action. By utilizing web scraping, we were able to compile a comprehensive database of teacher job postings in Washington State from late 2021 through 2022, updated twice a week.
Our analysis of the scraped data confirmed its accuracy by aligning with state administrative hiring data from the same period. It also provided new insights into teacher demand, revealing that job postings for special education and English language learner teachers remain open for significantly longer periods than those for elementary school teachers. Schools serving underrepresented minority students have greater hiring needs, with specialized openings remaining unfilled for extended periods.
This new source of data has broad applicability at the state and district levels, offering targeted and timely information for policymakers and education leaders. While states often respond to teacher shortages by increasing the overall supply of educators, ongoing analysis of job postings data can provide a detailed picture of demand and support precise strategies to address staffing needs where and when they are most needed.
Counting Teachers Isn’t Enough
The traditional policy response to teacher shortages has been to recruit more teachers. However, this approach, coupled with high turnover rates, often results in ongoing recruitment efforts that do not consistently meet actual staffing needs. The Education Commission of the States advised its members in 2022 that addressing shortages is less about recruiting teachers in general and more about finding and retaining the right teachers in the right subjects for the right schools.
Current data systems typically focus on tracking teachers rather than identifying specific subject areas and schools in need. A survey conducted by the National Council on Teacher Quality in 2022 found that only a quarter of state education agencies tracked unfilled teaching jobs statewide, with even fewer tracking this data at the school or subject-area level. This lack of detailed information hinders policymakers’ ability to target resources effectively.
Many states are exploring new pathways to increase the supply of teachers, including partnerships with fast-track credentialing companies that place trainees in paid classroom positions early in their training. While these programs aim to address shortages, they may not always target the most acute staffing needs, such as the demand for special education teachers.
In contrast, the State of Illinois has taken a demand-driven approach through the Teacher Vacancy Grant Pilot Program, allocating up to $135 million over three years to help districts attract, hire, and retain teachers based on their number of unfilled teaching positions. By using job postings data gathered by the Illinois State Board of Education, the program aims to address staffing needs effectively.
The limitations of existing state data systems also impact our understanding of market forces and teacher hiring patterns. For example, studies in Florida and Massachusetts have shown that economic conditions can influence teacher hiring rates, but the lack of direct evidence on demand for teachers makes it challenging to assess the impact of these trends on teacher quality.
A Test of Web Scraping
To test the effectiveness of web scraping in capturing detailed data on open teacher jobs and identifying specific challenges in the teacher labor market, we conducted a study in Washington State. By collecting job postings for classroom teaching positions in 2021 and 2022, we were able to analyze differences in staffing challenges by subject area and district type.
Our study focused on scraping job postings from 242 school districts that accounted for 98.5% of all public school students in Washington. By scraping district sites twice a week, we were able to compare the accuracy of job postings data with state hiring data and identify variations in staffing challenges.
The data revealed that elementary school teaching positions filled more quickly than other subject areas, while openings for special education teachers remained open the longest. Job postings for English language learner teachers also stayed open for an extended period, highlighting the specific challenges in filling these positions.
Analyzing job postings by student demographics showed that schools with a higher percentage of underrepresented minority students posted more job openings throughout the year compared to schools with a lower percentage. These schools also experienced greater hiring needs for special education and English language learner teachers, indicating disparities in teacher demand based on student demographics.
Web scraping technology offers a cost-effective and timely method of collecting data on school staffing needs. By tracking job postings regularly, policymakers and education leaders can gain valuable insights into hiring trends and tailor strategies to address specific shortages in subject areas and school types.
Differences by Subject Area and Student Demographics
Our findings highlight the complex nature of teacher shortages, hiring, and turnover, which vary by subject area, district, and school. By tracking open job postings over time, we can observe variations in the number of postings and the duration they remain open, providing a more nuanced understanding of staffing challenges.
Job postings data reveal that elementary teaching positions fill more quickly than other subject areas, while special education and English language learner positions remain open for longer periods. Schools with a higher percentage of underrepresented minority students have greater hiring needs, particularly for specialized positions, underscoring the importance of addressing equity in teacher recruitment.
The data also show that schools serving underrepresented populations face unique challenges in hiring credentialed teachers, with students in need of special education or English language learner services experiencing longer periods without instruction. This disparity underscores the need for targeted interventions to address inequities in teacher demand and supply.
New Insights and Opportunities
Web scraping technology provides a cost-effective and efficient way to gather detailed data on teacher job postings, offering timely insights into school staffing needs. By tracking job postings regularly, policymakers can make informed decisions to address shortages in specific subject areas and school types.
The timeliness of job postings data is crucial, especially when compared to traditional reporting methods that may lag behind current needs. By utilizing scraped data, policymakers can access up-to-date information on school hiring trends and tailor interventions to meet immediate staffing challenges.
Job postings data can also serve as a measure of policy impact, providing valuable information on the effectiveness of initiatives aimed at addressing teacher shortages. By analyzing job postings data, policymakers can assess the impact of programs like federal relief funding on school staffing and make informed decisions about future investments.
While job postings data do not capture all aspects of teacher shortages, they offer a valuable complement to traditional data collection methods. By leveraging web scraping technology, states can target resources more effectively, address shortages in specific subject areas, and promote equity in teacher recruitment.
In conclusion, teacher shortages are a multifaceted issue that requires tailored solutions based on subject area and school type. Web scraping technology offers a valuable tool for gathering real-time data on school staffing needs, providing policymakers with the information needed to address systemic inequities in public schools effectively. By analyzing job postings data, we can gain valuable insights into teacher demand and supply, informing targeted strategies to attract and retain qualified educators where they are needed most.
