Hopefully, your PhD research will make an impact by contributing to the knowledge in your field. While this kind of impact is difficult to measure substantively, more or less useful estimates of impact come from data on how often and how widely research is cited. However, quality in research is not necessarily captured by citation-based metrics, and recent initiatives reflect a growing interest in a broader and fairer basis for research assessment. On this page you will learn about
- practical ways of improving your impact by making your research visible, accessible and understandable;
- bibliometric indicators, such as impact factor and h-index;
- criticisms voiced against bibliometric impact measures and their application;
- the Declaration on Research Assessment (DORA) and the recent movement to find fair and robust ways of evaluating research that do not rely on impact factors;
Improve your impact
Good research + high visibility = Your best chance to make an impact
Being aware of how academic performance is evaluated allows you to make informed decisions and devise strategies to build and document your impact, thereby improving your career prospects. Our general advice is to make your work visible, accessible and understandable.
Make your work visible to other researchers:
- Prioritize journals that are known and read in your field. Your work is then more likely to be cited by your peers. When looking for a publishing venue, consider journals (or publishers) that are indexed in databases used in your field.
- Showcase your publications:
- Create your scholarly online identity by using ORCID and Google Scholar (Google Scholar profiles) and link your publications to your profile(s).
- Cite your previous work to further disseminate your research. However, do not overdo this, and make sure that you always stay in line with the topic discussed.
- Share your work in social networks. Commercial social networks specifically designed for academics exist, such as ResearchGate and Academia.edu. Impact measures based on usage and social network mentions are emerging and available on many websites, e.g. on journal and database websites. Work that has been shared and spread is more likely to get cited. However, be aware of copyright issues when considering sharing your research on these platforms.
- Make sure your work is added to the research register at your institution via NVA. These data are typically used for performance measures, so state your name and institutional affiliation on your publications in a correct and consistent manner.
- Engage and participate in scholarly debates in society, e.g. in the press. News coverage may also increase scholarly attention. Use your institutional communications department for aid in public or media communication.
- Collaborate with other researchers. In general, collaboration can benefit your career by increasing your production. Co-publishing may also imply borrowing some limelight from more renowned co-authors who are read and cited regularly.
Make your work accessible to other researchers by adopting open science practices:
- Publish open access. Openly available articles or books are easily spread and cited. Publishing open access is encouraged and supported at many institutions and mandated by many funding bodies. If you have published in a subscription journal, archiving a version of your manuscript in an open repository will make it markedly more accessible (NVA).
- Share your research data along with your publication. This strengthens your research and makes your findings replicable and verifiable.
- Making your publications and your data openly available (as far as possible), is likely to increase your chances of having your work cited (McKiernan et al., 2016).
Make your work understandable to other researchers:
- Use informative, memorable descriptions including key words in the title and abstract.
- Place your findings in a larger context by citing the work of other researchers in your field.
- Publish in English for a wider international distribution, if this aids in reaching a wider relevant audience. If you have written in your native language, consider republishing it in English, or publish a summary of your main findings in an international journal.
Citations and evaluation
Although it is difficult to pinpoint a connection between peer judgments and citation frequencies, citations are still perceived as indicators of quality and are used for e.g.
- benchmarking universities
- scholarship and employment decisions
- political decisions regarding research funding
- exploring research fields and identifying influential works and research trends
Citations must be handled carefully when evaluating research. Remember also that citation data vary from database to database, depending on the coverage of content of the database.
PROBLEM: THE MATTHEW EFFECT IN SCIENCE
To those who have, shall be given...
When sorting a set of publications by the numbers of citations received, the distribution shows a typical exponential or skewed pattern. Works which have been cited are more visible and are more easily cited again (vertical tail in figure), while other works remain hidden and are hardly ever cited (horizontal tail in figure). This phenomenon is referred to as the Matthew effect in science.
What is the problem with skewed distributions? Skewed patterns make it difficult to determine an average citation count. Different approaches may be applied, see the figure.
- Mean citation count: Long vertical or horizontal tails distort the average value. The impact factor is an example of this type of average.
- Citation median: Long vertical or horizontal tails distort the median value. For example, a long tail of rarely cited publications results in a low median value, while a minority of highly cited publications is ignored.
- H-index: The h-index is an alternative average value. It is designed to compensate for the effect of long tails.
Impact Factor
The most well-known measure of journal rank is the journal impact factor (often abbreviated to IF or JIF). It was developed in order to select the journals to be included in the Science Citation Index (Garfield, 2006). The impact factor is a measure of how often articles in a particular journal have been cited on average, in a given year. The central idea is that the impact factor indicates the journal's relative influence among journals within the same subject category.
A journal's impact factor is based on citation data from the Web of Science database, owned by Clarivate Analytics. If your institution has access to Web of Science, you can look up a journal's impact factor and related statistics there.
CRITICAL REMARKS: IMPACT FACTOR
Over the years, critical remarks have been raised towards the impact factor. You can read more about some of the criticism here.
No verifiable relation to quality
The impact factor is associated with a journal's prestige and is sometimes considered a proxy for the scientific quality of the work it publishes. Unfortunately, there is no verifiable association between journal impact factor and reasonable indicators of quality (for an overview of the relevant research, see Brembs, 2018).
Invalid measure of central tendency
The journal impact factor is, roughly, a measure of an average value, or mean. Such means are sensible indicators of central tendency if the distribution of values is symmetrical. However, citations to scholarly articles are not symmetrically distributed. Most published articles receive few, or even no, citations, while a small number of articles become very highly cited. This skewness means that a journal's impact factor is a poor predictor of the citation count of any given single article published in that journal (Seglen, 1997; Zhang, Rousseau & Sivertsen, 2017).
Field dependency
Because the impact factor is field dependent (due to different citation conventions), only journals within the same scientific field are comparable. Nevertheless, the impact factor is sometimes inappropriately used to compare journals from different fields (Adler, Ewing & Taylor, 2009).
Anglo-American bias
The pool of selected journals has a strong Anglo-American bias. Influential journals written in other languages are rarely captured by Clarivate Analytics Journal Citation Reports.
General coverage problems
Not all scientific journals are indexed in major scientific databases, although they may publish research with high relevance at local levels or contexts.
Unintended use
The impact factor is not only used for ranking journals according to their relative influence, as initially intended, but also for measuring the performance of individual researchers. Given the skewness of citation distributions described above, this becomes a misuse of the measurement. The use of the impact factor when applied to individual researchers has been criticized by a broad scholarly community, not least the co-creator of the Science Citation Index, Eugene Garfield, himself.
Typically, when the author's bibliography is examined, a journal's impact factor is substituted for the actual citation count. Thus, use of the impact factor to weight the influence of a paper amounts to a prediction, albeit coloured by probabilities. (Garfield, 1999)
Manipulation
The impact factor can be manipulated. Editors may influence the value of their journal's impact factor by writing editorials containing references to articles in their journal (journal self-citations). In addition, references given in the editorial count to the numerator, while editorials do not count towards the denominator. By definition the denominator only consists of citable articles, and editorials are not regarded as such.
Incomplete references
References provided in articles may be incomplete and incorrect. Incorrect references are not corrected automatically and therefore are not added to the citations. This fact influences the value of the impact factor and other citation indicators such as the h-index.
Alternative journal indicators
In order to compensate for some of the weaknesses of the impact factor (field dependency, inclusion of self-citations, length of citation window, quality of citations), efforts have been undertaken to develop better journal indicators. More advanced metrics are usually based on network analyses, such as the SCImago Journal Rank (SJR) and the Source Normalised Impact per Paper (SNIP), both based on data from Scopus. While such measures arguably do a better job of ranking journals, they are still only applicable to journals and should not be used to evaluate output at the individual researcher level. For that purpose, the h-index, introduced below, is better suited.
The h-index
The h-index is a measure of the total, citation-based, impact of a researcher. It combines scientific production (number of publications) and impact (number of citations).
When exploring the literature of your research field, the h-index may give you an idea of the impact of individual researchers and research groups. You can retrieve the h-index from e.g. Web of Science, Scopus and Google Scholar.
Some institutions or agencies may expect you to state your h-index when applying for a scholarship, project funding or a position. If you do, you should state from which source you retrieved it.
CALCULATING SOMEONE’S H-INDEX
In the table below, an author's publications (labelled a through j) have been rank ordered according to their number of citations.
| Rank | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Publications | c | i | a | g | f | h | d | j | b | e |
| Citations | 90 | 45 | 20 | 10 | 4 | 4 | 3 | 3 | 1 | 1 |
Table 2: Publications sorted by decreasing number of citations.
Counting from left to right, we reach four publications (c, i, a, and g) before the number of citations becomes less than the count. This author's h-index is, therefore, equal to 4.
Naturally, there is no need to manually calculate someone's h-index in this manner. It can easily be retrieved from Web of Science, Scopus or Google Scholar. Be aware, however, that a researcher’s h-index may differ from database to database. It will typically be lower in Web of Science and Scopus than in Google Scholar, since the two former are more restrictive in terms of journals and publication types indexed.
CRITICAL REMARKS: H-INDEX
Using a three-year citation window we find that 36% of all citations represent author self-citations. However, this percentage decreases when citations are traced for longer periods. We find the highest share of self-citations among the least cited papers. (Aksnes, 2003)
The h-index alone does not give a complete picture of the performance of an individual researcher or research group. The h-index underrepresents the impact of the most cited publications and does not consider the long tail of rarely cited publications. In particular, the h-index cannot exceed the total number of publications of a researcher. The impact of researchers with a short scientific career may be underestimated and their potential undiscovered. This brings us back to the problem of the Matthew effect in science.
h-index rules of thumb:
- The h-index is comparable only for authors working in the same field.
- The h-index is comparable for authors of the same scientific age.
- The h-index differs between databases, depending on the coverage in the individual database.
- The h-index depends on your institution's subscription time range. The h-index may underestimate researchers’ impact if their older publications are not included.
- The h-index is manipulable. Exaggerated use of self-citations may influence the h-index and result in an inflated value.
Alternatives to citation-based assessment
Evaluating research and researchers is not easy. While citation-based impact metrics, such as the journal impact factor, are convenient and have been popular, they have serious limitations and drawbacks as research assessment tools. Below follow descriptions of some initiatives advocating a broader and fairer basis for research assessment, and what this entails. In general, these initiatives encourage you to find alternative ways to describe and document any impact your work may have that is not captured by citation-based impact metrics.
The Declaration on Research Assessment (DORA)
The Declaration on Research Assessment (DORA) was published in 2013 and represents a global, cross-disciplinary initiative requesting better methods for evaluating research and researchers. The number of signatories is growing, and it has been signed by the Research Council of Norway and several other Norwegian research institutions. Multiple significant research funders are also among its signatories (e.g. the Wellcome Trust).
DORA signatories commit to not using journal-based metrics (such as the impact factor) in decisions regarding funding, hiring and promotions. Furthermore, they commit to being explicit about the criteria for making such decisions, to consider the value and impact of all research outputs (e.g. datasets and methods innovations), and to expand the range of impact measures to include such things as influence on policy and practice.
The Leiden Manifesto
This manifesto, created by the Centre for Science and Technology Studies (CWTS) at Leiden University, builds on the DORA initiative and outlines ten principles for the responsible use of bibliometrics in research assessment. Published in 2015, it was a response to concerns about the misuse and over-reliance on quantitative metrics in assessing research quality and impact.
The manifesto states that evaluation and evaluation criteria must be relevant to the specific goals of a research project and to the research missions of institutions, groups, and individuals. It also stresses that research may have a particular importance to local contexts, which is not always reflected by international metrics.
The manifesto also highlights the need for transparency in data collection and analysis and the importance of qualitative expert assessment of research output. The manifesto warns against using journal impact factors as a proxy for the quality of individual research articles.
The Coalition for Advancing Research Assessment (CoARA)
Like the Leiden Manifesto, CoARA has emerged as an initiative meant to address shortcomings in the traditional research assessment based on metrics. CoARA has established a new paradigm for evaluating research and its contributors, where the Agreement on Reforming Research Assessment is central. This agreement outlines ten commitments guiding the transition toward improved research assessment practices.
The above-mentioned initiatives represent an important development. Arguably, they imply that you may benefit from considering ways in which you can describe and provide documentation of any influence your work may have that is not captured by citation-based impact metrics. That said, all researchers should understand how indicators used to evaluate them work and how such indicators can be misused.
Altmetrics (alternative metrics)
Since traditional bibliometrics as presented above (impact factor, h-index) are sometimes considered too rigid, “altmetrics” (alternative metrics) have emerged as an alternative. Similar to citation data, altmetric data are available from various providers, and you might have encountered them when accessing articles you have discovered.
Altmetric data can provide a measure of how scholarly publications are shared, used and discussed online. This can give an indication of the influence a publication may have in online newspapers and news services, blogs, social media, Wikipedia articles and Q&A services, and so on.
References
Adler, R., Ewing, J., & Taylor, P. (2009). Citation statistics. Statistical Science, 24(1), 1-14.
Aksnes, D. W. (2003). A macro study of self-citation. Scientometrics, 56(2), 235-246. https://doi.org/10.1023/A:1021919228368
Brembs, B. (2018). Prestigious science journals struggle to reach even average reliability. Frontiers in Human Neuroscience, 12, Article 37. https://doi.org/10.3389/fnhum.2018.00037
Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161, 979-980. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1230709/
Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295, 90-93. https://doi.org/10.1001/jama.295.1.90
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature (London), 520(7548), 429–431. https://doi.org/10.1038/520429a
Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569-16572. https://doi.org/10.1073/pnas.0507655102
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., . . . Yarkoni, T. (2016). How open science helps researchers succeed. eLife, 5, Article e16800. https://doi.org/10.7554/eLife.16800
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ, 314(7079), 498-502. https://doi.org/10.1136/bmj.314.7079.497
Zhang, L., Rousseau, R., & Sivertsen, G. (2017). Science deserves to be judged by its contents, not by its wrapping: Revisiting Seglen's work on journal impact and research evaluation. PLoS ONE, 12(3), Article e0174205. https://doi.org/10.1371/journal.pone.0174205
