Limitations of bibliometrics

Bibliometrics such as h-index or times cited each provide one kind of insight into the reach and effect of work. However, because of how research is done and how scholarly publishing works, they are not necessarily accurate and thorough.

“Any metric can be gamed, especially singular metrics such as citation counts.” — Christine L. Borgman, Big Data, Little Data, No Data: Scholarship in the Networked World (2015)

Scope of the tool

Common tools such as Web of Science and Scopus provide a particular view of the bibliographic universe. This view may be limited by:

  • formats of materials included (e.g., journals, conference papers, books, book chapters);
  • subject breadth (what falls within the subject scope of the tool?);
  • subject depth (how much of a particular discipline’s scholarly output is included?);
  • geographic coverage (do publications emanating from particular regions get included?);
  • language coverage; and
  • depth of backfiles (e.g., how far back are citations tracked?).

In all cases, it is important to ask what a particular tool includes and what it excludes. There is no one article database that includes all articles ever published and which can analyze all the relations between them. (Publishing open access is one way of fighting this, and York University Libraries can help.)

Disciplinary differences

Some disciplines, particularly in the humanities, rely more heavily on particular formats for scholarly output, especially books and book chapters. These are not tracked well in tools such as Web of Science and Scopus, nor is the subject breadth and depth for humanities and social science journals as broad and deep within these tools. Therefore scholars who write books may have their impact, as measured by these methods, misrepresented.

Publications in some disciplines do not rely as heavily on citations to other work. Thus, the reduced number of citations interconnecting publications may not properly capture the impact of a publication. Moreover, these metrics do not necessarily work well for creative works and may not reflect local cultural practices.

Researchers who publish in journals that serve subdisciplinary specialties not well-covered by the common tools will not be accurately tracked by the bibliometrics that common tools produce.

At the very least, disciplinary particularities do not allow for cross-disciplinary comparisons of impact.

Inherent limitations of the metrics

A scholar may produce few “units” of scholarly output (books, journals), yet this small body of work may be seminal for a particular field and yield tremendous impact on a discipline’s scholarship. Standardized metrics, such as the h-index, will have difficulty accounting for such situations.

Large research teams in some disciplines may produce dozens or hundred of research papers, each with dozens or hundreds of authors. Members of these teams will often demonstrate very high impact metrics that may not accurately reflect their individual prominence within the field.

This quote, from bibliometrics researchers, says it well:

“Quantitative metrics are poor choices for assessing the research output of an individual scholar. Summing impact factors, counting citations, tallying an h-index, or looking at Eigenfactor™ Scores (described below)—none of these methods are adequate compared with what should be the gold standard: reading the scholar’s publications and talking to experts about her work.” — Bergstrom, C. T., West, J. D., & Wiseman, M. A. (2008). The EigenfactorTM Metrics. The Journal of Neuroscience, 28(45), 11433–11434.