This page will look much nicer in a browser that supports CSS, or with CSS turned on.

Uncertain Principles

Physics, Politics, Pop Culture

Tuesday, August 23, 2005

Luck is Not a Problem-Solving Technique

With the summer drawing to a close, another academic year is ramping up. One of the earliest indicators of the madness that is to come is the release of the US News and World Report college rankings.

There are lots of problems with these rankings, but they're very popular with students and parents, so colleges take them very seriously. Attempts to improve in the rankings come in many forms, from the sincerely well-intentioned to the astonishingly cynical, but any college or university official who tells you they're not acutely aware of their institution's standing is flat-out lying to you. (My current employer jumped up four spots this year, and there was much rejoicing.)

Of course, any successful scheme inspires imitators, and there are now dozens of copycat rankings. Some of these are merely an attempt to cash in, while others attempt to highlight some perceived deficiency in the US News ratings. The latest entry in the latter category is the Washington Monthly's College Rankings (via Kevin Drum, of course). Like the good squishy liberal types that they are, they have the best of intentions:

How much more important, then, is it for taxpayers to know that their money—in the form of billions of dollars of research grants and student aid—is being put to good use? These are institutions, after all, that produce most of the country's cutting-edge scientific research and are therefore indirectly responsible for much of our national wealth and prosperity. They are the path to the American dream, the surest route for hard-working poor kids to achieve a better life in a changing economy. And they shape, in profound and subtle ways, students' ideas about American society and their place in it. It seemed obvious to us that these heavily subsidized institutions ought to be graded on how well they perform in these roles, so we set out to create the first annual Washington Monthly College Rankings. While other guides ask what colleges can do for students, we ask what colleges are doing for the country.

While these are admirable goals, they end up with a ranking scheme that is, if anything, even sillier than the US News rankings, at least when it comes to liberal arts colleges.

For the sake of full disclosure, I should note that I'm writing this largely because my alma mater takes a big hit in the rankings. As they note:

And the traditional prestige schools didn't all benefit from the Washington Monthly ranking system. Williams, which U.S. News ranks as the top liberal arts school in the country, wound up at #14 on our list, one slot below Presbyterian, largely because of its weak service numbers.

But let's take a look at how they get those numbers:

We determined the Community Service score by measuring each school's performance in three different areas: the percentage of their students enrolled in the Army or Navy Reserve Officer Training Corps; the percentage of their students who are currently serving in the Peace Corps; and the percentage of their federal work-study grants devoted to community service projects.

Now, this represents a good-faith effort to construct something useful out of publicly avaiable data. The end result, though, is pretty heavily biased against private schools in rural areas. Two out of the three measures are inherently more difficult for a rural school to do well on (the Peace Corps being the exception).

Take the ROTC component. Very few liberal arts colleges have enough participating students to support their own ROTC organization, so students who are interested in that are obliged to piggy-back on other institutions. For example, I've had a couple of students in class who were taking part in ROTC programs, and they've all needed to go to either Albany or Troy (~20 min drive) for the required drills. That's not too big a deal if you're in an urban area, but from Williams, that would require a drive of around an hour, over some roads that get pretty treacherous during the winter. In principle, ROTC is an option, but in practice, it'd be awfully difficult to manage.

And then there's the community service issue. Community service is a good thing (though somewhat incidental to the actual purpose of a college or university), but by its nature, it's highly dependent on the nature of the local community. If you're in New York or Boston, there are endless opportunities for meaningful community service projects. If you're in Berkshire County, the pickings are slimmer, just because there aren't anywhere near as many people to serve.

That's compounded by the normalization method-- they measure "the percentage of their federal work-study grants devoted to community service projects." Not only is the numerator of that fraction necessarily smaller for rural colleges, the denominator is also bigger for the elite schools. The higher tuition charged by the top-rank private colleges is going to require a larger amount of work-study funding, pushing the percentage down (you can see this in the university rankings, too, where 6.5 of the top ten instituions are state schools (I'm counting Penn as private, and Cornell as half-and-half)).

If you look at the whole list, you can see the systematic effects. Look at the top five schools: Wellesley is in Boston, Wesleyan is between Hartford and New Haven, Bryr Mawr is in Philly, Harvey Mudd is in LA, and Fisk University is in Nashville. Look at the prominent schools that that are in the middle of nowhere: Williams dropped 13 spots, Middlebury 9, Grinnell 10, Hamilton 14, and Carleton 25. The only anomaly is Colby College in Maine, which jumped up 11 spots despite being way out in the sticks (and ought to be commended for it).

The irony here is that I don't actually disagree with their conclusions. Is Williams a service-oriented place? Not really, no. I don't think I knew anybody who would've opted for ROTC if only it had been less of a drive, and the bulk of the community service work I was aware of was court-ordered.

But as I tell my students, getting the right answer by the wrong method doesn't get you full credit.

(A couple of disclaimers, here. First, the Washington Monthly people are not completely oblivious to the crappiness of their metrics-- they provide a list of data that they would prefer to use, but that isn't publicly available. As they put it, "...we have tried to abide by the best principles of social science and used the best data available to generate the closest possible measures of the qualities we value."

(Also, it's entirely possible that I'm way off base in my analysis, and it's the Peace Corps thing that's sinking Williams's ranking, not any of the factors related to its location. I have no way to tell, as the "best principles of social science" evidently don't include making the component data available on the Web.)

Posted at 8:38 AM | link | follow-ups | 3 comments

Monday, August 22, 2005

Two Weeks Without You, Thought I'd Forget

We've hit that point in the summer-- finally-- when my research students have left, and I can actually take a little bit of a vacation. Yes, I know, one of the advantages of my job is supposed to be getting the summers off, but, you see, I don't have tenure. Thus, the summer is almost exactly as hectic as the academic year.

My nerves are stretched to just about the breaking point, though, so I'm taking this week off. Later in the week, Kate and I are heading up to Quebec City for a few days (suggestions of must-see attractions and activities are welcome in the comments), but I'm not going to work at all this week, letting me finally deal with some of the jobs that have been going undone around the house.

One of the before-vacation tasks is to clean out those Bloglines feeds that I've marked as things to comment on. So, here's a list of miscellaneous stuff:

First, Arcane Gazebo reports some nifty results ina fairly comprehensible manner. It's already been linked by some other people, but it's worth pointing out as a fairly readable explanation of some fundamental quantum-mechanical research. More like this, please.

On the political side, I think I'm going to stop posting comments on current events, and just link to Timothy Burke when he posts. He keeps saying eminently sensible things that are basically what I want to say, but am too clumsy to actually get down in text. He's got good pieces recently on "Intelligent Design" and political theatrics-- now, if only he had an RSS feed.

For another side of blogging, sennoma points to the blog Tricks of the Trade, offering, well, little tricks that experts in some field know, that might be useful to ordinary people. It offers no less than five different ways to start a fire in the woods, along with tricks to keep people from stealing your pens, and a possible way to make the elevator go faster. There's some neat stuff.

Speaking of tricks of the trade, props to Matt McIrvin for his Really Basic Photo Editing With GIMP 2.0. I've been assigned to be the department webmaster this year, and his tips have helped a great deal with cleaning up some of the digital pictures we have on the page. I've also been doing some cargo-cult JavaScript, with limited success. When I get the whole thing finished, I'll post a link.

Matt also linked to streaming video of Feynman giving some well-known lectures. I haven't watched it yet, as Kate's torrents of bits have dragged the connection speed down under 1 kB/s, but I'm posting this here to remind me to look at the damn thing.

On the multimedia front, Nanopolitan has a link to famous people talking about Einstein. Technorati is a wonderful thing when it works.

Finally, something silly: the proprietor of Conrdoggerel is offering an interpreting service:

Given that I am a male and that most of my readers are females, I thought I might offer my insights into the male psyche as a service to women in the blogosphere who are confused by a man or men in their lives. Let me list my qualifications:

I have played: football, basketball, baseball, golf, bowling, horseshoes, poker, quarters.

I can match or exceed most of his qualifications (add to the above "I have played" list: "rugby," "beer pong," and "with fire"), but it'd probably be a bad idea to go through the whole list. I don't have tenure, but I would like to have tenure someday.

Maybe then, I'll get a real summer vacation.

Posted at 10:23 AM | link | follow-ups | 7 comments

ΔxΔp ≥ h / 4 π

My stuff
What's with the name?
Who is this clown?
Does he know what he's talking about?
Archived Posts
Index of Physics Posts
RSS, version 0.91
The Library of Babel
Japan Stories

Δ E Δ t ≥ h / 4 π

Other People's Stuff

AKMA's Random Thoughts
Arcane Gazebo
Arts and Letters Daily
Boing Boing
Chronicles of Dr. Crazy
Confessions of a Community College Dean
Cosmic Variance
Crooked Timber
Brad DeLong
Diary de la Vex
Drink at Work
Easily Distracted
Electron Blue
John Fleck
Grim Amusements
David Harris's Science and Literature Hellblazer
In the Pipeline
Invisible Adjunct
Izzle Pfaff
Knowing and Doing
The Last Nail
Learning Curves
The Little Professor
Making Light
Malice Aforethought
Chris C. Mooney
Musical Perceptions
My Heart's in Accra
Michael Nielsen
Not Even Wrong
Notional Slurry
Off the Kuff
One Man's Opinion
Orange Quark
The Panda's Thumb
Perverse Access Memory
Political Animal
The Poor Man
Preposterous Universe
Pub Sociology
Quantum Pontiff
Real Climate
The Reality-Based Community
SciTech Daily
Sensei and Sensibility
Talking Points Memo
Through the Looking Glass
Unmistakable Marks
Unqualified Offerings
View From the Corner of the Room
What's New
Whiskey Bar
Wolverine Tom
Word Munger
Yes, YelloCello
Matthew Yglesias

Book Stuff

Book Slut
Neil Gaiman
The Humblest Blog on the Net
Pam Korda
Outside of a Dog
Reading Notes
Seven Things Lately
The Tufted Shoot
Virtual Marginalia
Weasel Words
Woodge's Book Report


ACC Hoops
College Basketball (2.0)
Dave Sez
Hoop Time 3.0
The Mid-Majority
Set Shot
Tuesday Morning Quarterback

Δ N Δ Φ ≥ 1 / 2


75 or Less Album Reviews
Rotten Tomatoes
The Onion A.V. Club

Geek Stuff

Annals of Improbable Research
Astronomy Picture of the Day
Britney Spears's Guide to Semiconductor Physics
The Comic Book Periodic Table
MC Hawking's Crib
The Museum of Unworkable Devices
Myths and Mysteries of Science
The Onion
Physics 2000
Sluggy Freelance
Web Elements
Physics Central (APS)
This Week's Finds in Mathematical Physics

Useful Stuff

Web Design Group

While it is my fervent hope that my employers agree with me about the laws of physics, all opinions expressed here are mine, and mine alone. Don't hold my politics against them.

Weblog posts are copyright 2003 by Chad Orzel, but may be copied and distributed (and linked to) freely, with the correct attribution. But you knew that already.

If you use Internet Explorer, and the text to the right cuts off abruptly at the end of this column, hit "F11" twice, and you should get the rest of it. We apologize for the inconvenience.

Powered by Blogger Pro and BlogKomm.

Steelypips main page.