Wednesday, July 14, 2010

Reading research papers 101

Excellent piece from Daniel Drezner. Below is his recommendation. Read the full discussion here.
1)  If you can't read the abstract, don't bother with the paper.  Most smart people, including academics, don't like to admit when they don't understand something that they read.  This provides an opening for those who purposefully write obscurant or jargon-filled papers.  If you're befuddled after reading the paper abstract, don't bother with the paper -- a poorly-worded abstract is the first sign of bad writing.  And bad academic writing is commonly linked to bad analytic reasoning. 
2)  It's not the publication, it's the citation count.  If you're trying to determine the relative importance of a paper, enter it into Google Scholar and check out the citation count.  The more a paper is cited, the greater its weight among those in the know.  Now, this doesn't always hold -- sometimes a paper is cited along the lines of, "My findings clearly demonstrate that Drezner's (2007) argument was, like, total horses**t."   Still, for papers that are more than a few years old, the citaion hit count is a useful metric.
3)  Yes, peer review is better.   Nothing Megan McArdle wrote is incorrect.  That said, peer review does provide some useful functions, so the reader doesn't have to.  If nothing else, it's a useful signal that the author thought it could pass muster with critical colleagues.  Now, there are times when a researcher will  bypass peer review to get something published sooner.  That said, in international relations, scholars who publish in non-refereed journals usually have a version of the paper intended for peer review. 
4)  Do you see a strawman?  It's a causally complex world out there.  Any researcher who doesn't test an argument against viable alternatives isn't really interested in whether he's right or not -- he just wants to back up his gut instincts.  A "strawman" is when an author takes the most extreme caricature of the opposing argument as the viable alternative.  If the rival arguments sound absurd when you read about them in the paper, it's probably because the author has no interest in presenting the sane version of them.  Which means you can ignore the paper. 
5)  Are the author's conclusions the only possible conclusions to draw?  Sometimes a paper can rest on solid theory and evidence, but then jump to policy conclusions that seem a bit of a stretch (click here for one example).  If you can reason out different policy conclusions from the theory and data, then don't take the author's conclusions at face value.  To use some jargon, sometimes a paper's positivist conclusions are sound, even if the normative conclusions derived from the positive ones are a bit wobbly.  
6)  Can you falsify the author's argument?    Conduct this exercise when you're done reading a research paper -- can you picture the findings that would force the author to say, "you know what, I can't explain this away -- it turns out my hypothesis was wrong"?  If you can't picture that, then you can discard what you're reading a a piece of agitprop rather than a piece of research. 
7)  Fraudulent papers will still get through the cracks.  Trust is a public good that permeates all scholarship and reportage.  Peer reviewers assume that the author is not making up the data or plagiarizing someone else's idea.  We assume this because if we didn't, peer review would be virtually impossible.  Every once in a while, an unethical author or a reporter will exploit that trust and publish something that's a load of crap.  The good news on this front is that the people who do can't stop themselves from doing it on a regular basis, and eventually they make a mistake.  So the previous rules of thumb don't always work.  The  publishing system is imperfect -- but "imperfect" does not mean the same thing as "fatally flawed." 

Dani Rodrik on the ideology of markets


"Unlike economists and politicians, markets have no ideology. As long as they make money they do not care if they have to eat their words.  They simply want whatever “works”—whatever will produce a stable, healthy economic environment conducive to debt repayment. When circumstances become dire enough, they will even condone debt restructuring—if the alternative is chaos and the prospect of a greater loss.
This opens up some room for governments to maneuver. It permits self-confident political leaders to take charge of their own future.  It allows them to shape the narrative that underpins market confidence, rather than play catch-up.
But to make good use of this maneuvering room, policymakers need to articulate a coherent, consistent, and credible account of what they are doing, based on both good economics and good politics. They have to say: “we are doing this not because the markets demand it, but because it is good for us and here is why.”
Their storyline needs to convince their electorates as well as the markets. If they succeed, they can pursue their own priorities and maintain market confidence at the same time."
More here. Krugman adds more here.