Co-authored with Phil Fernbach.
In last week’s post, we presented an analysis of the captions for The New Yorker’s Cartoon Caption Contest #281 (which Roger Ebert won). We hoped that the analysis would help generate a general set of principles that would be useful to folks who want to win future contests. Our takeaway:
Try for subtlety, elegance and originality. And definitely do not use exclamation points!
On Generalizability
Bob Mankoff, The New Yorker’s Cartoon Editor, was rightly critical of the generalizability of an analysis based on one contest:
…. I can say with certainty that we shouldn’t compromise our research ethics by generalizing the results beyond this particular caption contest to all the other 283 contests, or, for that matter, to non-caption contest New Yorker cartoons, or to humor in general. That would be a grave mistake which would only be justified if we could make a lot of money out of it.
(For Wireds’ take on the exchange see Cracking the Code of The New Yorker’s Cartoon Caption Contest)
Bob puts forth a few intuitively appealing counter examples that stand in contrast to our analysis. For example, contest #147 effectively uses an exclamation point.
Just as Bob warns us about generalizing from one contest, we warn against using a few counter examples selected purposefully from the other 280 contests.
To test the generalizability of our findings we suggest taking a random sample of 30 or more contests and see if our results replicate. A replication would suggest that we have discovered a guiding principle (though not helpful in all cases, would be useful if applied consistently). A failure to replicate could occur for at least two reasons:
1) The appeal of the caption is largely defined by the nature of the cartoon. And in this way, our analysis is of limited use to people trying to win the contest.
2) The appeal of the caption is largely defined by the whims of the editors. And in this way, our takeaway would be useful if not for the judgmental biases of the editorial gatekeepers.
Of course, both reasons are likely to be true to some extent. The suggestion that the selection process is in some way biased is not a controversial idea in the field of behavioral decision theory. A large literature demonstrates that judgmental processes can be influenced by extraneous factors – ranging from order effects to fatigue to whether it is a sunny day.
One way to differentiate the two reasons would be to see if captions that adhere to our principles are more likely to win the voting. That is, we would conduct an analysis that compares the winners (by popular vote) to second and third place across all the contests.
On the Objective of the Analysis
Bob’s response provided an additional challenge to our analysis: The principles are obviously true. The analysis is merely reiterating what everyone already knows.
…I do want to thank Pete and Phil for doing this. While the results are, in fact, not shocking at all, a lot of useful science is just showing that what you think is true, is true. And sometimes, in doing that you find out as Mark Twain said that, “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.”
While we agree with Bob to a point, that there can be (some) scientific value in providing data that the obvious is true, we don’t see that as what we are doing. For one, many caption entrants are violating these principles suggesting there is some value in pointing them out (e.g. to the 824 entrants who used an exclamation point in their caption).
More importantly, while each finding in isolation may be unsurprising in retrospect, we believe there is a richer way to think about and utilize these findings: Intuitively, we know a good caption when we see it but it is a holistic judgment; we don’t go through a set of important attributes, tick them off and then chuckle when a caption fits the criteria. A caption is not good because it is short. It is not good because it uses surprising words. It is not good because it uses little punctuation. A caption is good because it is witty, subtle and elegant; it makes you think about a cartoon in new way. Shortness, word frequency, punctuation and concreteness are not causes they are symptoms.
The way to think about how to use these principles is that they should serve as evidence of whether your caption is as good as it can be not as a recipe for generating a caption. If your caption is overly long it may suggest there is a more pithy way to express your idea. Or perhaps your idea is too complicated to express in a good caption. If your caption has punctuation, is it doing work or is it superfluous? If your caption refers directly to items in the cartoon, is it doing so in a novel way or is the joke obvious and tired?
The fact that great captions can be long, use frequent, concrete words and punctuation is completely consistent with this perspective. We are reminded of Jennifer Grey: The Dirty Dancing star had the nasal equivalent of an exclamation point. Yet face perception, like caption evaluation is a holistic judgment. And in the context of her face, the nose worked! Indeed, a nose job is often blamed for the unfortunate trajectory of her post-Dirty Dancing career. Will removing an exclamation point negatively affect your caption? It might…