A new study by an independent researcher has found that coaching for college admission tests has only a small effect, despite the claims of companies and private tutors who have turned test preparation into a multimillion-dollar industry.
The study's data, from a national sample of more than 14,000 students, indicated that the average gain from coaching was no more than about 20 points on the 1,600-point SAT test. The study found coaching produced similarly small improvements in English and math for students taking the ACT, a test used by many colleges that do not use the SAT. Coaching actually seemed to have slightly lowered scores on the reading part of the ACT.
The findings support the long-standing contention of the College Board, which sponsors the SAT, that test coaching has little effect. But the study, reported in the current issue of Chance, a magazine of the American Statistical Association, had several flaws.
For one thing, it did not differentiate between intensive and expensive preparation courses that may last for months, and short, even one-day, preparation courses. Officials of major preparation companies said this failing called the results into question.
Also, the study compared students who chose coaching with those who did not, rather than randomly assigning students to one group or another.
Still, some experts on educational testing, like Stephen Klein, a senior research scientist at the Rand Corp. who studies educational testing, called the work "very important" and convincing.
More than 2 million high school students take the admissions tests each year, and 10 percent to 12 percent of them sign up for commercial coaching programs, which may cost $700 to $3,000 for a course.
The study's author, Derek C. Briggs, a doctoral student in education at the University of California at Berkeley, based his conclusions on an analysis of data in a national survey by the Department of Education that follows a representative sample of students from eighth grade through high school and beyond. Briggs said he had no connection to any testing service or coaching concern.
Until now there has been no large national study on the subject and few studies done outside the auspices of the College Board or the companies that sell test preparation courses.
Previous studies with designs like Briggs' have found similarly small effects from coaching. Other studies have sometimes reported far more substantial benefits in coaching, but they also have had serious design flaws, said Nan Laird, a biostatistics professor at the Harvard School of Public Health, and other academic statisticians examining the data.
Laird said most of the studies that showed big gains simply compared test scores before and after students were coached. But they did not include, for comparison, the scores of students who were not coached but who simply took the tests again. Retaking itself usually results in higher scores.
While not arguing with the study itself, testing companies said the results did not apply to them.
Seppy Basili, vice president for learning and assessment at Kaplan Inc., a leading test preparation company, said there were big differences between test preparation courses. "What we've seen over the past 15 years is this huge increase in weekend courses and one-day courses," Basili said. Kaplan's courses, which cost $800, last three months. "This whole notion of grouping commercial courses with this broad brush causes a problem for us."
He added that the company's surveys of its students showed that they increased their SAT scores by an average of 120 points.
Though Briggs' study avoided many drawbacks of previous ones, it was not ideal, said James Robins, a professor of epidemiology and biostatistics at the Harvard School of Public Health, who cited the fact that students were not randomly assigned to groups.
As a result, Robins said, he would not be surprised if the true coaching gain was as much as 40 or 60 points — or if it was even less than 20 points.
Robins also said that at highly competitive colleges, any advantage, even 30 or 40 points in an SAT score, could make a difference between being accepted and being rejected.