Here is CPS’s analysis of a 2020 report by Papay et al., which is being used by Commissioner Riley and DESE to bolster a proposal to raise the scores students must get on the MCAS to get a high school diploma. We sent it to Sec. Peyser, Commissioner Riley and the members of the Board of Elementary and Secondary Education yesterday. (We also sent it to all of the legislators who signed a letter to the BESE opposing the proposal and to the members of the legislature’s Joint Education Committee.)
July 13, 2022
Dear Commissioner Riley, Secretary Peyser, and members of the Board of Elementary and Secondary Education,
In anticipation of your discussion and vote on the proposal to raise the MCAS competency determination standard, we at Citizens for Public Schools offer this analysis of the cautions in the 2020 John Papay et al. Lifting All Boats? Report, included in this message and attached as a PDF document.
We hope you will find it helpful in your decision making process to consider the following analysis. Please feel free to reach out with any questions.
Sincerely,
Lisa Guisbond
Executive Director
Citizens for Public Schools
At its July 2022 meeting, the Massachusetts Board of Elementary & Secondary Education (BESE) is considering raising the passing score on MCAS tests. This proposal is primarily based on one finding in a June 2020 report by John Papay et al., titled Lifting All Boats? Accomplishments and Challenges from 20 Years of Education Reform in Massachusetts: students with higher MCAS scores have higher earnings and higher rates of graduating from a four-year college than students with lower MCAS scores.
However, there was a reason why the report title, Lifting All Boats, ends with a question mark. The report includes significant concerns and cautions, buttressed by a key finding in an earlier Papay report on MCAS. These should be reviewed by BESE as they consider the commissioner’s proposal. All quoted text in the following points #1-6 and #8 is taken directly from the Lifting All Boats? report:
1) MCAS is a flawed and probably the wrong test.
“While MCAS tests assess students’ academic skills, they do not measure most dimensions of social skills. This asymmetry creates a problem in assessing the progress of Massachusetts public education. It is possible that accountability tied to MCAS tests has led some schools to respond in ways that do not develop useful academic and social skills, for example by narrowing the curriculum or focusing on test-taking strategies.” P.4
The authors include “useful academic…skills” in addition to social skills, such as communication, problem solving, reasoning and analysis, collaboration, etc., the very skills that colleges and businesses cite as most essential for success.
2) The finding that students with higher MCAS scores have better earnings is not causal; hence, BESE has rushed to judgment.
“We should note that all of the research presented here is descriptive in nature. We cannot make causal claims about the relationships we present. For example, while we document that students with higher MCAS scores have better earnings, we cannot conclude that raising MCAS scores definitively causes higher earnings.” P.5 (in notes)
3) You can’t make accurate extrapolations from some of their statements, for example:
“In 2011, 89% of students at the 25th percentile of the test distribution graduated from high school, compared to 97% at the 75th percentile. Similarly, only 19% of students at the 25th percentile graduated from a four-year college, compared to 68% at the 75th percentile. We see quite similar patterns in ELA.” P.7-8
There are at least two problems with this statement: First, presenting graduation from a four-year college as the holy grail, the desired end result for all students, is problematic. Many lower-scoring students might be more interested in vocational schools and the trades than a four-year college. Yet, BESE’s own vocational school admissions regulations deny entry to a disproportionate percentage of low-income students, students of color, English Learners, and students with disabilities by enabling vocational schools to rank order applicants by grades, discipline record, attendance, guidance counselor recommendations, and interviews, criteria that have evidence of race, language, disability, and income bias. Second, the finding reflects a blindness to the challenges of low-income students (who are inclusive of the majority of English Learners and students of color) in financing a four-year college degree in a nationwide college system built to favor privileged and more resourced students.
4) MCAS score increases since 2008 (only five years after high-stakes MCAS took effect) are due to scale drift and/or score inflation – thus, MCAS is having little impact as it is and is doing more harm than good:
“In both ELA and mathematics, the average 8th grade NAEP scores of Massachusetts public school students increased through 2005-06 (corresponding to 10th grade test takers in 2007-08) but have remained quite flat thereafter. This suggests that the increases in average MCAS mathematics and ELA scores until approximately 2008 represented real gains in academic proficiency, while the increases since then have been almost entirely due to scale drift, score inflation, or a combination of the two.”
See point #7 on the harm of MCAS for low-income students.
5) The gaps they cite represent 56%, or over half, of MA public school students.
“…students from low-income families are less likely to earn a four-year college degree than students from higher-income families who have the same MCAS scores.” P.11 “In all cases, English learners, Black students, and Hispanic students score substantially lower on average than their peers in both math and ELA.” P.19
High-needs students in MA (which encompass low-income, EL, and disabled students, and low-income and ELs disproportionately represent students of color) represent 56% of all students.
6) The four-year college graduation gaps have increased rather than narrowed:
“The percentage of low-income students who graduated from a four-year college increased from 10% for 2003 MCAS test-takers to 18% for those who took the 10th grade MCAS in 2011. The comparable college graduation rates for higher income students are 38% and 52%. Thus, the gap in the graduation rate widened from 28 to 34 percentage points over an eight-year period…. In virtually all cases,…the gap for four-year college graduation has increased.” P. 19
These increases are substantial – 16 percentage points for low-income, 13 points for Blacks, 15 points for Latinx, and 6 points for ELs.
In a separate 2008 Papay et al. paper, The Consequences of MCAS Exit Examinations for Struggling Low-Income Urban Students, the authors find:
7) The harmful effects of MCAS for low-income students are real:
“...for low-income urban students on the margin of passing, failing the 10th grade mathematics examination reduces the probability of on-time graduation by eight percentage points. Furthermore, failing the 8th grade mathematics examination reduces by three percentage points the probability that low-income urban students stay in school through 10th grade. Importantly, we find no effects for suburban students or wealthier urban students. Given that 26% of low-income, urban students who just pass the exam do not graduate on time, this effect is quite substantial.”
8) The 2020 report by Papay et al. does not explicitly endorse raising the MCAS passing standard and makes no specific recommendations to address the problems and concerns identified. In fact, the report cautions against overemphasizing the MCAS.
“We must be clear – improved MCAS scores are not the goal, but should reflect improvements in underlying capacities and skills. …Too much emphasis on the test, rather than the skills it is designed to measure, can result in higher scores without improving the academic and social skills of students in the Commonwealth.” P.25.
In sum, DESE went cherry-picking to bolster their proposal to raise the passing MCAS scores and, as a result, is putting the cart of MCAS before the horse of improving educational access and quality.
