Showing posts with label SAT. Show all posts
Showing posts with label SAT. Show all posts

2011-01-25

Selection Biases in Real World Data

So a web-page depicting the average SAT score by state has been making the rounds a little bit in Iowa. Apparently Iowans are so smart! But something seemed fishy about the data...oh, right the participation rate is 3% in Iowa, and in general states with a lower participation rate have better scores.



Now, if you take the ACT data for 2010, Iowa is around 15th, but with a much higher participation rate.



Look at Maine! Maine is ranked dead last, 51st, on the SAT list, but 5th on the ACT list, up there with a lot of East Coast states whose students didn't do so well on the SAT list. New York is 46th on one list and 4th on the other.

Why the difference in Iowa's rank? And why the huge difference in Maine's rank?

Biases in the test-takers is likely to be at work. ACT has its mothership in Iowa City, meaning that Iowa is likely to be pushing the ACT for political-economic ties. In my experience, only seasoned test-takers in Iowa take the SAT. The ACT is "good enough" for most people, but because of the additional practice and the inherent variance of the tests, you can get a higher personal best simply by more tests (personal note from an MIT grad: I took each test 6 times, once per year from 7th grade to 12th grade). Thus it is likely that Iowans hoping to get into a selective school are more likely to take the SAT. A similar effect is probably at work with the ACT in states like Massachusetts --- lower participations rates in these states are being caused by forces which select for the best test-takers and smartest students.

Making the SAT difficult to take in Iowa might help the state look the best in the SAT rankings, even though the ACT is HQ'ed in Iowa City. But most of all, reading too much into improperly gathered rankings is dangerous.