Selection Biases in Real World Data

So a web-page depicting the average SAT score by state has been making the rounds a little bit in Iowa. Apparently Iowans are so smart! But something seemed fishy about the data...oh, right the participation rate is 3% in Iowa, and in general states with a lower participation rate have better scores.



Now, if you take the ACT data for 2010, Iowa is around 15th, but with a much higher participation rate.



Look at Maine! Maine is ranked dead last, 51st, on the SAT list, but 5th on the ACT list, up there with a lot of East Coast states whose students didn't do so well on the SAT list. New York is 46th on one list and 4th on the other.

Why the difference in Iowa's rank? And why the huge difference in Maine's rank?

Biases in the test-takers is likely to be at work. ACT has its mothership in Iowa City, meaning that Iowa is likely to be pushing the ACT for political-economic ties. In my experience, only seasoned test-takers in Iowa take the SAT. The ACT is "good enough" for most people, but because of the additional practice and the inherent variance of the tests, you can get a higher personal best simply by more tests (personal note from an MIT grad: I took each test 6 times, once per year from 7th grade to 12th grade). Thus it is likely that Iowans hoping to get into a selective school are more likely to take the SAT. A similar effect is probably at work with the ACT in states like Massachusetts --- lower participations rates in these states are being caused by forces which select for the best test-takers and smartest students.

Making the SAT difficult to take in Iowa might help the state look the best in the SAT rankings, even though the ACT is HQ'ed in Iowa City. But most of all, reading too much into improperly gathered rankings is dangerous.

ACM Survery of Graphs

ACM has an interesting survey of graph types. I don't like most of them, *especially* graph types like the stacked graphs in 1B that introduce correlations into the visual representation of the data that aren't in the data. The splom in Figure 2C, however, is a very nice representation that I'd never seen before.

Rants on Media Continued: Misleading Graphs

USA Today has a very misleading graph on the change in the American, Catholic population. The graph is meant to show the change in regional catholic population, but I found two problems between the graph and its mission. For the first problem, take a second and think about the circled states:



California and Texas have large increases in their Roman Catholic populations, and Florida has a sizable increase. These three states comprise 25% of the population of the US. This graph, which was supposed to represent growth in various regions, instead portrays a very inaccurate picture of overall growth. Yahoo! News breaks it down correctly:

Nationally, Catholics remain the largest religious group, with 57 million people saying they belong to the church. The tradition gained 11 million followers since 1990, but its share of the population fell by about a percentage point to 25 percent.

Would you have guessed, after looking at the graph above, that the Catholic population in the US increased? Or that the percentage of Catholics in the population dropped one percentage point, which is a drop in the portion of the population of roughly four percentage points (26 to 25)? By focusing on the four percentage point drop in the population portion data (even making this drop appear larger than it is), the graph misses both overall growth and the one percentage point drop in the overall population. USA Today's article text doesn't even mention the overall growth.

Forgoing the big picture to explain a less important point is the second problem with the graph. How would you visually show the information from the quote above? The more I have to create visual data, the more I'm struck by how much a sentence can communicate.

(Yahoo! News even links to the study. I still stand by my last rant, however.)

Daylight

As I go slowly insane while analyzing data, I find myself wanting to analyze more data. Thanks to the efforts of the U.S. military, I can fulfill these caprices. I have concocted a few graphs to illustrate daylight in the different places where I've lived. These graphs do not include the effects of weather. Note that southern CT is not included, as it has roughly the same longitude as Cedar Rapids, IA.

First up is a graph of minutes of daylight per day versus time of year.


Delft gets over 1,000 minutes of daylight per day in the middle of the summer. Or, rather, the cloud-layer over Delft gets this daylight. Boston and Cedar Rapids only have small differences in daylight per day.

I was wondering how much less daylight Delft gets per year than Iowa, and to my surprise, Delft gets more daylight!




The second graph is a zoom of the previous graph. As you can see, Delft gets roughly one more day of daylight each year than Cedar Rapids and Boston.

I am looking forward to the days following the Equinox here. One thousand minutes of daylight. Sounds like a novel.