29 November 2024
Christopher Steven Marcum, Open Science Advocate, Washington, DC, USA
https://orcid.org/0000-0002-0899-6143
For most research positions in academia tenure and promotion schemes rely heavily on the applicant’s peer-reviewed publication record. Despite nuances around the world in how academic promotion functions – with differences in gradation, involvement of organized labor, and technical requirements between countries and institutions – the “publish or perish” adage is nearly universal as a requirement for job security in a university.
The reliance on publications as the currency for tenure and promotion has created significant inequities in academia and has resulted in perverse incentives in the scholarly publication market that favors quantity over quality of research, with significant adverse consequences for the integrity of published research that threatens public trust in science. Recognizing that job security and institutional recognition are desirable outcomes for the careers of most academic researchers, a growing reform movement aims to restore the scientific integrity of published research, reduce inequities in promotion schemes, and reward academics equitably for all their scholarly outputs.
As one of the leading organizations in the global movement, the Coalition for Advancing Research Assessment (CoARA) coordinates across local, regional, and national academic assessment organizations in an effort to shift the evaluation culture away from traditional quantitative metrics and towards a more holistic approach. Signatories of the coalition agree to provide action plans that respect local requirements while aligning with the broader mission of the coalition. CoARA has two dozen signatories from UK – and over a hundred more across the world – including universities, funders, and scholarly societies.
Here are three simple and free things departments, universities, and funders can do to make their recognition processes more equitable based on recommendations from this reform movement:
1. Jettison journal titles from CVs and letters of reference.
The culture within the academy places a premium on the “prestige” of the journals in which an author’s manuscripts appear. And while a sense of personal accomplishment may very well justify the prestige value of a journal, it is an ecological fallacy for any particular article to inherit that prestige for the purposes of merit recognition.
Instead, the merits and integrity of the research should be self-evident to an author’s peers. The problem with journal prestige is that it induces biases into performance reviewers’ mental models, consciously or subconsciously, that favor articles appearing in more prestigious journals even if those appearing in less prestigious journals – or are not even published in journals – have objectively equal or even greater scientific merit. This is particularly problematic and inequitable for the many classes of researchers that, for systemic and structural reasons, are less likely to publish in the most prestigious journals (especially for women, researchers with external responsibilities, [1] [2] and those that live in the global South, for instance).
Recognizing the biases that journal titles can induce on performance reviewers, the Howard Hughes Medical Institute issued a policy that categorically disallows any reference to journal titles for funding application and renewal packages. By doing this, HHMI researchers can expect to benefit from a more even competitive environment that is free from prestige bias.
2. Substitute impact factors with narrative impact statement.
Citation-based personal impact factors, such as the H-Index, are well-known to be unfair, inaccurate, and inequitable when used as measures of meaningful contributions to science and society or as a proxy of academic reputation. They are also easily manipulated by self-citation or transactional citation, giving authors a perverse incentive for malfeasance.
When incorporated into the academic reward system, impact factors can disadvantage early career researchers, researchers who publish in narrowly read fields, and researchers who have structural barriers to publishing in widely read outlets. They also do not account for many of the significant contributions that advance science well beyond any impact that article citations might capture.
Academics make many unrewarded contributions to society. For example, in the United States, researchers might contribute to notice and comment on Federal regulations that improves the rulemaking process and have their research cited into a federal rule or regulation. Scholarship might be cited by findings in Congress or Parliament that would not be indexed by the major citation indices. Academics are also regularly called by courts as expert witnesses, and often testify before Congress and Parliament to provide their subject matter expertise before policymakers. Some have founded movements that engage communities with science and evidence-based policymaking. All of those examples are critically important contributions that have significant impact beyond the few people that will ever read an article and are yet never captured by an H-index or equivalent. These non-publication outputs are also discussed below. [3]
As an alternative to these metrics, review criteria should rely on narrative impact statements from both promotion applicants and their outside referees. Many funders already require impact statements – those, however, typically only focus on the impact of incremental contributions of science manifested through publications or the anticipated but not yet realized impact of a grant. Because these requirements can often be satisfied as a simple “check the box” exercise, true cultural reform that rewards impact requires a more serious structural change. This is the primary motivation behind the changes in assessment coming to the UK through the REF 2029 – higher education institutions will need to demonstrate commitment to people, culture, and environment and how their faculty are incentivized and rewarded for aligning their outputs with this commitment.
Guidance to applicants, referees, and reviewers on the content of narrative impact statements should encompass all of the ways that applicants for academic promotion contribute to society. One funder, Fonds National de la Recherche in Luxembourg, has taken a forward approach by developing a free narrative CV template for use by their grant applicants to “…reduce bias in evaluation through responsible research assessment, as well as a drive towards a more inclusive research culture…”.
3. Provide parity to non-publication output for values-based assessments.
One of the advantages of relying on publications as a currency for tenure and promotion is that articles are easy to count and demonstrate as outputs. Beyond publications though, academics produce a wide array of outputs. Some, like data, code, and patents, are also easy to quantify. Others, like mentorship, service contributions, and community engagement are not as easy. Like publications, however, all of these outputs should be characterized by the value they bring to society, which requires a qualitative assessment of whether the productivity of an applicant for tenure and promotion reflects the values of their programs and fields.
By recognizing and rewarding these diverse contributions, academic institutions can incentivize activities that are essential for the advancement of knowledge, innovation and the overall health of the academic ecosystem. Giving parity to non-publication outputs in tenure and promotion supports social values that demand transparency, equity, security, and integrity in research. Summaries of these outputs would quite naturally fit into the narrative impact statements described above, rather than simply enumerated into a quantitative metric. One incredibly useful resource to help institutions move away from an evaluation paradigm that relies on metrics that are easy to count to one that relies on value-alignment can be found in a free toolkit created by a diverse group of open science advocacy organizations.
Summary
Reforming academic reward systems to be more open and equitable will require a move away from the traditional emphasis on publication metrics. By jettisoning journal titles, substituting impact factors with narrative impact statements, and providing parity to non-publication outputs, institutions can incentivize a broader range of scholarly activities that better reflect the diverse values of the academic community and society. When taken together, [4] these three simple steps for academic institutions and funders represent a path towards tenure and promotion reform that can be taken with a little effort to eventually replace the publish or perish adage with produce and prosper.
Disclaimer
The views expressed in this editorial represent only those of the author and do not necessarily reflect the opinions or positions of his employer or associations.