Director’s Reflections…Do Impact Factors Matter? The San Francisco Declaration on Research Assessment

Barbara EpsteinHow to measure the impact of research publications is a perennial challenge. One of the most common yardsticks is the journal impact factor (JIF), a formula which measures the frequency with which a journal’s hypothetical “average article” was cited in a particular year. Developed about forty years ago by Eugene Garfield and Irving H. Sher, the JIF can be useful in evaluating a journal’s relative importance as compared with other publications in the same field. Through the years, many academic institutions have come to use JIF to evaluate individual faculty and researchers, though it was clearly designated to measure overall journal quality, rather than the impact of individual articles in a particular journal. Critics have identified many weaknesses, including editorial manipulation, human error and variations among scientific disciplines.1

Last month, a group of editors and publishers of scholarly journals, brought together by the American Society for Cell Biology (ASCB), jointly issued a set of recommendations—known as the San Francisco Declaration on Research Assessment (DORA)—to improve how scientific research output is assessed by funding agencies, academic institutions and the public. DORA’s original signers included 155 individuals and 82 organizations, including the American Association for the Advancement of Science, ASCB, the Howard Hughes Medical Institute, and the Wellcome Trust. At the time this article was written, the list of signers has grown to 7,715 individuals and 289 organizations.

Though recognizing that peer-reviewed research papers are likely to remain the central research output for the foreseeable future, DORA acknowledges that the importance of other types of research outputs, such as datasets, reagents, and software, will likely grow in the future. The recommendations, aimed at funding agencies, academic institutions, journals, organizations that supply metrics, and individual researchers, have a number of common themes, including:

  • The need to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations.
  • The need to assess research on its own merits than on the basis of the journal in which the research is published.
  • The need to capitalize on the opportunities provided by online publication.2

DORA has its own Web site, hosted by ASCB, with the text of the declaration, list of signers, and a growing list of articles and editorials commenting on DORA. Time will tell whether DORA itself will have an impact on how research is assessed.

1. Weiss, P. “Academic Career Confidential: The Truth about Journal Impact Factors.” HSLS Update 11 no. 6, (December 2006): 4-5.

2. “DORA: San Francisco Declaration on Research Assessment.” American Society for Cell Biology, http://am.ascb.org/dora/ (cited on June 17, 2013).