Fellow Scientopian DrugMonkey has blogged a perfect storm of a discussion on impact factor and glamour science. Click on over and read the comments (warning: your head may explode). This argument will sound familiar to most readers. Basically, everyone knows that the impact factor (IF) can be gamed by journals. IF reflects some sort of average citation rate for a journal; it says nothing about the quality of any given paper. Some people make the point that IF keeps the measurement of productivity from being solely a pub count. Others add that IF is imperfect, but it's "what we have."
At Science Online I attended a discussion of Alternative Metrics or altmetrics:
As the volume of academic literature explodes, scholars rely on filters to select the most relevant and significant sources from the rest. Unfortunately, scholarship’s three main filters for importance are failing:
- Peer-review has served scholarship well, but is beginning to show its age. It is slow, encourages conventionality, and fails to hold reviewers accountable. Moreover, given that most papers are eventually published somewhere, peer-review fails to limit the volume of research.
- Citation counting measures are useful, but not sufficient. Metrics like the h-index are even slower than peer-review: a work’s first citation can take years. Citation measures are narrow; influential work may remain uncited. These metrics are narrow; they neglect impact outside the academy, and also ignore the context and reasons for citation.
- The JIF, which measures journals’ average citations per article, is often incorrectly used to assess the impact of individual articles. It’s troubling that the exact details of the JIF are a trade secret, and that significant gaming is relatively easy.
I hoped that the discussion would provide a gentle introduction to the concept of altmetrics. My hopes died, and I felt adrift during the session. I have played with some of the new measures on the altmetrics site. I get what these researchers want to do; I just have not figured out how each measure fits into a bigger picture. [I do appreciate more of the discussion now.]
For a kinder, gentler introduction to the topic, I recommend a piece in the current issue of The Chronicle of Higher Education that profiles Jason Priem, a graduate student in library sciences at the University of North Carolina at Chapel Hill. He helped develop Total-Impact, an altmetrics site that tracks information as it is discussed across the web. He discusses the general concept of the site as well as its current limitations (hey, it's still in alpha).
The internet disrupts traditional publishing; we no longer need to fit the scientific record to the dead-tree world of volumes and issues and page numbers. This shifting paradigm is dragging metrics along, potentially crushing IF in the process.