The admissions test for Boston Latin and the city's two other exam schools has not been shown to accurately predict students of color are likely to do in high school, according to the test-maker's own study.
The Educational Records Bureau creates the ISEE, or Independent Schools Entrance Exam, that the Boston Public Schools uses for exam school admissions.
"We are looking to collect more data from more schools so that we can have more students from different demographic groups to do analysis by gender, by race/ethnicity," said Rochelle Michel, the executive director of the New York-based nonprofit.
Rochelle confirmed that the ISEE's validity study, which determines the extent to which the test predicts high school achievement, had a sufficient sample size only for white students.
The study included 13 independent schools from across the country, which the study acknowledges is "a very small group relative to the population of schools that administer the ISEE on an annual basis." Those 13 schools had a sufficient number of white students but not minority students to draw conclusions about achievement.
The study also notes that "no conclusions can be drawn about the relationship between ISEE scores and high school academic outcome measures by ethnicity."
Lauress Wise, a principal scientist at the Human Resources Research Organization based in Alexandria, Virginia, said the district is making high-stakes decisions based on a test that might not predict the exam school performance of children of color.
"You really should do the study more fully than than was done here," he said. "Students, because of the low test scores, are being denied an opportunity that would actually advantage them considerably."
Wise said the Educational Research Bureau hasn't analyzed the test "to show that it's fair to students of different ethnic and cultural backgrounds." He said it's possible that black and Hispanic students who get lower ISEE scores would perform well in exam schools.
"You could resolve the issue of fairness if you had adequate sample sizes and were able to demonstrate that the relationships between the scores on the ISEE and outcomes of interest were the same" across ethnic groups, he said. "In other arenas, you get over- or under-predictions, so minority students with the same ISEE scores may actually do better on grade point averages and (test) scores."
Admission to the exam schools is based on two factors, GPA and ISEE scores, each with equal weight.
Michel said individual test questions were examined for differences across racial and ethnic groups.
"We look at students across groups who are performing similarly, and then look at how they’re performing on the different items, and then (evaluate) any items where one group may perform different than another," Michel explained.
But Laura Hamilton, a senior behavioral scientist and associate director at RAND Education, said that kind of analysis isn't good enough on its own.
"The assumption there is that those two groups are equivalent on the underlying ability that the test is designed to measure, so if you see an item where one group scores a lot higher than the other, that would suggest that there’s a problem with that item," she said. "But if there’s a bias or some other source of difference across the whole test, you don’t see that when you do that analysis because you’ve matched on total test scores. You need other ways to look at bias at the test level."
Harvard's Rappaport Institute released a report earlier this month that found high-achieving black and Hispanic students do worse on the ISEE than their white and Asian peers with similar MCAS scores.
The study found that black and Hispanic students in the top 25 percent of MCAS scores did worse on the ISEE than Asian and white students in that group. Black and Hispanic students scored twelve percentiles lower on the ISEE, despite similar MCAS scores.
As a response to the report, Boston Public Schools has decided administer the ISEE in every sixth grade classroom next year. Interim Superintendent Laura Perille said the district hopes that expansion will increase the rates of black and Hispanic students taking the test.
Melanie Rucinski, who co-authored the Harvard report, said that intervention may not have the desired effect.
"The district is planning on increasing access to the ISEE and changing the test locations and test times to make it more accessible to students," she said. "Even among students who do take the ISEE, even among these high achieving students, we do see gaps in the scores that are correlated with student race."
The district paid the Educational Research Bureau $1.2 million over the last three years of administering the ISEE to Boston Public Schools students. Students take it in sixth grade to get into Boston Latin, Boston Latin Academy and the John D. O’Bryant School of Mathematics and Science.
Mayor Marty Walsh said the district hasn’t ruled out changing the test.
“We're looking at it now," he said in an interview with WGBH News. "It's it's one of the many things that is being considered by the school department.”
At the same time, Walsh said he’s reluctant to change the admissions process for the exam schools.
“By changing the standards of the school, I don't know if that's the proper way of going about it," he said. "Latin School and Latin Academy and John D. O'Bryant is not for every kid.”
The district signed a new contract with the Educational Research Bureau in July.
Our coverage of K-12 education is made possible with support from the Nellie Mae Education Foundation.
