Is there something odd about this paper?

Or cats and critical appraisal.

I wrote in the previous post about F.D.C. Willard, co-author of a paper on low temperature physics, and actually sole author of another in French a few years later.   Willard, as the lead author of the first paper, J.H. Hetherington admitted some years later, was in fact a cat, added to the paper as a reaction to editorial policy about the use of "we" when there was only one author.   

I used to do a drop in workshop for research postgraduates on critical appraisal, which I started after talking to one of the departments I worked with about whether there was a need for such a thing.  Because we never knew who would be there, and because there might be people from a wide range of disciplines, the session had to be fairly general.  I devised a critical appraisal checklist to work with any discipline in the (as it was then) Faculty, which covered medicine, biological sciences and psychology.

F.D.C. Willard (Chester to his family) was a way into the idea that everything is not always what it seems with a journal paper.   If Hetherington had not said, we would of course never know that there was anything unusual with this paper, and the unusual thing was not to do with the physics.  My point was that if we conduct a critical appraisal of a paper, we might well spot any unusual things.  As part of the workshop we then conducted a critical appraisal, using my checklist.  With participants potentially from such a wide range of disciplines, choosing a paper was difficult (and I also had to be able to understand it!).

So, we appraised this paper from the Annals of Improbable Research.  This quite often worked well, but there were times when the humour of it was difficult for some of the participants, coming as they did from a range of cultures.

So, how do you know that there is something unusual about a paper?

Here are some possibilities:

• A critical appraisal.  Part of this is of course to see if the paper applies to your studies, research or patient, but part of it is to examine the methodology.  There might be cases when the paper passes the critical appraisal test, but there is still something unusual.  Maybe one of the following will help:
  
• Look for an erratum, corrigendum or retraction (we talked about those in that critical appraisal session);
• Letters and other responses, if the journal publishes them.  The BMJ allows "rapid responses", some of which become letters in the journal, something that the journal has always published;
• Comments on the journal website.   I am not sure how many specialist journals allow this;
• Blogs and other discussions about the article;
• Media coverage - perhaps only after something odd has been discovered (other things that make it into the media are perhaps articles that are thought to be significant, ground breaking, or have eye catching press releases - I have no evidence for any of these statements!);
• Check a systematic review of the topic and see if the paper in question (assuming it is a type that could be included in one) is in or out, and check how the results compare to the others in the review and to the results of the review.

If there is none of this, then I think you would need to find other studies of the same thing and compare results.