Edited by Daniel Acuña, University of Colorado Boulder, Boulder, CO; received September 30, 2024; accepted March 18, 2025 by Editorial Board Member Mark Granovetter
August 4, 2025
122 (32) e2420092122
Edited by Daniel Acuña, University of Colorado Boulder, Boulder, CO; received September 30, 2024; accepted March 18, 2025 by Editorial Board Member Mark Granovetter
August 4, 2025
122 (32) e2420092122
Significance
Numerous recent scientific and journalistic investigations demonstrate that systematic scientific fraud is a growing threat to the scientific enterprise. In large measure this has been attributed to organizations known as research paper mills. We uncover footprints of activities connected to scientific fraud that extend beyond the production of fake papers to brokerage roles in a widespread network of editors and authors who cooperate to achieve the publication of scientific papers that escape traditional peer-review standards. Our analysis reveals insights into how such organizations are structured and how they operate.
Abstract
Science is characterized by collaboration and cooperation, but also by uncertainty, competition, and inequality. While there has always been some concern that these pressures may compel some to defect from the scientific research ethos—i.e., fail to make genuine contributions to the production of knowledge or to the training of an expert workforce—the focus has largely been on the actions of lone individuals. Recently, however, reports of coordinated scientific fraud activities have increased. Some suggest that the ease of communication provided by the internet and open-access publishing have created the conditions for the emergence of entities—paper mills (i.e., sellers of mass-produced low quality and fabricated research), brokers (i.e., conduits between producers and publishers of fraudulent research), predatory journals, who do not conduct any quality controls on submissions—that facilitate systematic scientific fraud. Here, we demonstrate through case studies that i) individuals have cooperated to publish papers that were eventually retracted in a number of journals, ii) brokers have enabled publication in targeted journals at scale, and iii), within a field of science, not all subfields are equally targeted for scientific fraud. Our results reveal some of the strategies that enable the entities promoting scientific fraud to evade interventions. Our final analysis suggests that this ability to evade interventions is enabling the number of fraudulent publications to grow at a rate far outpacing that of legitimate science.
Sign up for PNAS alerts.
Get alerts for new articles, or get an alert when an article is cited.
Over the last four centuries, the production of scientific knowledge has increasingly become a matter of state and societal importance. The “contract” between scientists and states can be summarized thusly: In exchange for creating new knowledge that is useful to the state and training a workforce able to use that knowledge, society supports scientists with rewarding careers, good salaries, and public recognition. The success of this contract has led to an extraordinary growth in the scale and scope of the scientific enterprise (1) and to its adoption across the world (2). Indeed, some studies suggest that the wealth of a nation is closely aligned with the amount (3, 4) and quality (5) of the research it produces.
The state-supported scientific enterprise can be idealized as a public goods game (6) with numerous and diverse stakeholders. Because of the increasing complexity of the knowledge being created and increased specialization, the system relies on the good-faith assumption of genuine contributions by all participants (7–10). Scientists rely on other scientists to disclose knowledge that can be built upon, on other scientists and on publishers for the screening of scientific studies, on publishers for the dissemination of their work and on funding agencies and universities for support. Universities and funding agencies rely on scientists for evaluating the work of their peers and on the state and society for their funding. Private-sector firms rely on universities to educate a knowledgeable workforce. The state and society rely on scientists to produce knowledge that will improve well-being and state security. Etzkowitz and Leydesdorff formalized certain aspects of this web of relationships in their ‘triple helix’ model of knowledge-based economic development (11).
The success of this model could be in jeopardy if some stakeholders fail to contribute fairly to the tasks assigned to them. Due to the increasing scale and scope of the scientific enterprise, the degree to which stakeholders contribute to the system is now increasingly evaluated by potentially misleading proxies (12, 13) such as the h-index (14), journal impact factor, university rankings, and scientific prizes. Nonetheless, these proxies have quickly become targets for evaluation of institutional and personal impact, resulting in increasing competition and growing inequality in how resources and rewards are distributed (15–20), which could leave the scientific enterprise more susceptible to defection (16, 21–23).
Scholarly defection occurs when there is a failure to make genuine contributions to the production of knowledge or to the training of an expert workforce while still benefiting from the contract. A 2002 survey of scientists funded by the United States NIH reported that 0.2% of mid-career researchers and 0.5% of early-career researchers admitted to falsifying research data in the previous three years (16). A systematic analysis of more than 20,000 articles published between 1995 and 2014 reported that 3.8% of these articles contained inappropriately duplicated images, with at least half of these cases suggestive of deliberate manipulation (24). We and others have also recently described a class of entities engaging in large scale scientific fraud, typically denoted “paper mills,” that sell mass-produced low quality and fabricated research articles (as described by Byrne et al. (25) and in a report by the Committee on Publication Ethics and the International Association of Scientific, Technical & Medical Publishers (26); also see SI Appendix). In a 2022–2023 survey of medical residents at tertiary hospitals in southwest China, 46.7% of respondents self-reported buying and selling papers, letting other people write papers, or writing papers for others (27). Some publishers report that up to 1 in 7 of their submissions are of probable “paper mill provenance” (26, 28). Agents for paper mills have also recently been reported to attempt to bribe journal editors (29, 30) and to “hijack” the entire editorial processes at some journals (31–33).
Studies of repeating public goods games teach us that, under some conditions, player contributions tend to decay over time and that contributions decrease substantially as the number of defectors increases (34). To discourage defection and sustain a collaborative system, public goods games must enforce mechanisms that disincentivize defection (35–37). To this end, the scientific enterprise has implemented several formal punishment mechanisms for defectors. Funding agencies can sanction individual researchers and universities with fines and exclusion from funding programs. Universities can sanction researchers with rescission of contracts. Journals can sanction authors with retraction of publications. Literature aggregators can sanction journals by removing them from their indices (deindexing) (38–40). These formal measures complement additional informal measures such as exclusion of defectors from scientists’ personal trust networks, shaming (41), and documentation of concerns on postpublication review sites (42, 43). However, the evidence suggests that these mechanisms have not yet been successful in stemming the tide of defections (24–33).
Here, we demonstrate that large networks of cooperating individuals and entities that produce scientific fraud at scale can be identified by the footprints they have left in the extant scientific literature. We identify groups of individuals who perform editorial and review work at journals and collaborate with authors to facilitate the publication of manuscripts that experts (including journal editorial staff) have linked to author or editor misconduct. We show that organizations producing fraudulent science are able to ensure publication of their manuscripts in journals from several publishers prior to their detection and removal. We also show that organizations mediating the production of manuscripts suspected of fraud evade existing science integrity and quality control measures, such as journal deindexing, and that they successfully target specific subfields while avoiding—or failing to target—closely related subfields. Finally, we show that the number of fraudulent publications is growing at a rate far exceeding the growth of legitimate publications.
Materials and Methods
We retrieved journal data from Clarivate’s Web of Science (WoS) (44), Elsevier’s Scopus (45), the National Library of Medicine’s PubMed/MEDLINE (46), and OpenAlex, an open-source “index of hundreds of millions of interconnected entities across the global research system” (47). OpenAlex aggregates and standardizes data from the now deprecated Microsoft Academic Graph (48) and Crossref (49), as well as ORCiD (50), Unpaywall (51), and institutional repositories. We considered a journal to be “actively publishing” in a given year if it published at least one “journal article” or “conference proceeding article” in that year in OpenAlex. Of 73,818 journals actively publishing in 2020, 1,489 (2.0%) had “conference,” “proceeding,” or “meeting” in the journal title. Of 5,629,023 articles published in 2020, 476,820 (8.5%) were labeled as conference proceeding articles.
Literature Aggregators.
WoS and Scopus evaluate journals requesting inclusion in their databases (i.e., indexing). Indexing of a journal can be revoked if the service finds reasons for concern with the editorial practices of the journal or if the service assesses a journal’s content to be of low quality. A journal may also be deindexed due to a name change, merger, or closing. We obtained lists of annually deindexed journals from WoS (current as of December 2021), Scopus (current as of March 2024) (52), and MEDLINE (last downloaded in February 2023) (53, 54). We also downloaded the 2020, 2021, 2023, and 2024 editions of the Early Warning Journal List produced by the Chinese Academy of Sciences in June 2024 (55).
Retractions and PubPeer Comments.
Author misconduct, such as reporting fake data or engaging in plagiarism, and editorial misconduct, such as lack of minimal quality control of submitted work, can result in the retraction of a study or in its reporting in postpublications sites. We obtained retracted articles from Retraction Watch (56) (downloaded on March 4, 2024). This corpus comprises 47,387 unique retraction records. We downloaded the Retraction Watch Hijacked Journal Checker on March 1, 2024. The PubPeer Foundation shared with us metadata and content for all comments made prior to February 1, 2024, on 105,325 articles bearing DOIs. This count does not include automated comments, such as those made by statcheck (57). Very few PubPeer comments are positive (43); most report on potential problems and many precede eventual retraction of the commented article (58).
Editors.
Some journals list the handling editor of every published study. A small subset—including PLOS ONE and Hindawi journals—also allows for bulk, programmatic access to article content and metadata. We downloaded metadata of articles published in PLOS ONE by November 8, 2023, (59) and of articles published in Hindawi journals by April 2, 2024, (60) (Dataset S3). We obtained full names of editors at PLOS ONE and Hindawi journals and assigned articles to the same editor in each journal only if the full names were an exact match. We classified an editor as “active” during the period between the earliest submission date and the latest acceptance date of published articles handled by them. Note that editor tenures are right-censored because we do not have access to articles currently under review.
We disambiguated article authors using ORCiD identifiers (50) but could not obtain ORCiD identifiers for 91.2% of named authors. In total, we identified 134,983 authors of articles published in PLOS ONE for whom we could obtain ORCiDs. This left 39.8% of articles without any identifiable author. ORCiD identifiers were similarly infrequent in Hindawi journals, as described in SI Appendix, Figs. S4–S13.
IEEE Conferences.
The name and year of IEEE conferences were inferred from the DOI of all “proceedings” articles published by IEEE and recorded in OpenAlex. Matching this DOI to “conference proceedings” articles in OpenAlex and excluding articles with specific front matter-related terms in the title (Dataset S4), we identified 2,294,067 articles published in 19,969 IEEE conferences since 2003. For 45 IEEE conferences taking place between 2009 and 2011, more than 10% of the published articles were retracted. This activity results in a large spike in retractions around these years. Although many conference proceedings publish front matter that names editorial staff, this information was not available to us in a structured format that would allow for disambiguation of editor identities and roles.
Duplicated Images.
We collected reports of image duplication made on PubPeer and then built a network where nodes represent articles and edges indicate sharing of an image (see SI Appendix for details). We identified connected components in this network and filtered out any component with fewer than 30 articles. Additionally, we removed any component where any of five randomly selected edges does not concern interarticle image duplication.
The Academic Research and Development Association (ARDA).
Unlike previously documented entities (61), ARDA’s website lists journals in which they can guarantee publication. We used the Internet Archive’s “Wayback Machine” (62) to construct a chronology of ARDA’s evolving “portfolio” of journals. We then matched named journals to their indexing records in WoS and Scopus and constructed a visualization of the time periods over which those journals were being used by ARDA and whether they were ever deindexed; see Dataset S5 for the list of journals.
In May 2023, we downloaded the complete archive of five journals listed on ARDA’s website (two of which are considered hijacked) and attempted to impute the nationality of authors from listed affiliations (SI Appendix). We successfully imputed nationality of authors for 13,288 of 20,638 recovered documents (64.4%, Dataset S6). Of these documents, more than half were from India (26.4%), Iraq (19.3%), or Indonesia (12.2%).
Suspected Paper Mill Corpus.
We compiled a corpus of suspected paper mill products by aggregating records from multiple corpora curated by experts (Datasets S1 and S2). Suspected paper mill products are typically identified using manual and automated methods that search for unexpected similarities and shared content spanning multiple articles and journals (SI Appendix). We downloaded metadata of articles containing tortured phrases from the Problematic Paper Screener (63) on March 21, 2024. Currently, this corpus of suspected paper mill products comprises 32,786 unique articles.
Results
Anomalous Patterns in the Editorial Handling of Problematic Publications.
There is a perception among many practicing scientists that scientific fraud is a rare phenomenon (64) resulting from the actions of isolated actors (65). Mounting evidence, however, suggests the possibility that fraud is a more pervasive phenomenon; that defectors target journals to facilitate the publication of fraudulent science at scale (29). To investigate the latter possibility, we analyzed data from several journals that report the name of the editor responsible for accepted manuscripts and test whether certain individuals are more likely to edit problematic articles than one would expect by chance alone.
One of the journals that discloses the handling editor, PLOS ONE, has published 276,956 articles since 2006; 702 have been retracted and 2,241 have received comments in PubPeer (Fig. 1A). It is visually apparent that the retraction rate and comment rate are not constant. We determined for each of 18,329 editors who have accepted articles for publication in PLOS ONE (Fig. 1B) the number of articles that they accepted for publication and the number that was eventually retracted for each publication year. We then used a Poisson binomial test for whether each editor accepted ultimately retracted or PubPeer-commented articles significantly more often than expected by chance alone, adjusting for the variable rates of retraction and commenting in time (66) (SI Appendix). Since we tested multiple hypotheses, we adjusted the significance level via the Benjamini–Hochberg procedure (67), controlling the false discovery rate (FDR) at 0.05.
Fig. 1.
Evidence of coordination between editors and authors for the publication of fraudulent science in PLOS ONE. (A) Number of articles published annually in PLOS ONE since its inception (Top row) and the annual number of articles retracted (second row), receiving PubPeer comments (third row), or having been accepted 30 or fewer days after submission (Bottom row). (B) Number of “active” editors and their tenures at PLOS ONE (Materials and Methods). Flagged editors tended to have overlapping tenures. (C) Detection of editors accepting an anomalous number of articles which were later retracted (second row), later commented in PubPeer (third row), or accepted after a short turnaround (Bottom row). We show each editor as a circle with an area proportional to the number of articles they accepted. At a false discovery rate (FDR) of 0.05, we identify 22 editors who accepted an anomalous number of articles later retracted, 33 editors who accepted an anomalous number of articles later receiving PubPeer comments, and 88 editors who accepted an anomalous number of articles after a short turnaround. (D) Detection of authors whose accepted articles are assigned at an anomalous rate to flagged editors for retractions (second row), PubPeer comments (third row), and short turnaround (Bottom row). At an FDR of 0.05, we flagged 21 authors for anomalous assignment of their accepted articles to editors flagged for retractions, 18 authors for anomalous assignment of their accepted articles to editors flagged for PubPeer comments, and 7 authors for anomalous assignment of their accepted articles to editors flagged for short turnaround. (E) Publication interactions among individuals flagged for retractions and/or PubPeer comments. We show every individual as a circle whose size is proportional to the number of accepted articles that they handled as editor (range of 1 to 852 articles). The black wedge inside the circle indicates the fraction of that editor’s accepted articles which were later retracted. Individuals flagged for retractions alone have a red boundary, individuals flagged for PubPeer comments alone have a blue boundary, and individuals flagged for both have a purple boundary. The individuals are connected by arrows that point from author to editor, with the width of each arrow proportional to the total number of articles for which that editor served for that author (range of 1 to 10 articles). Flagged individuals frequently handled each other’s submissions to PLOS ONE, especially within a group of editors serving from 2020 to 2023 (cluster on the Lower Right).
Fig. 1C shows that 22 editors accept articles that were retracted significantly more frequently than one would expect by chance. When considering articles that have received PubPeer comments, the number of flagged editors increases to 33. Using this group of flagged editors, we next investigated whether there are authors that appear to direct their articles more frequently to these flagged editors than expected by chance. We again used a Poisson binomial test, adjusting for the variable activity of flagged editors over time (66) (SI Appendix). Fig. 1D shows that, for the 8.8% of PLOS ONE authors we could identify unambiguously, 21 authors had papers more frequently assigned to editors flagged for retractions than one would expect by chance.* We also identify 18 authors who direct their articles to editors flagged for PubPeer comments more frequently than expected by chance.
For further verification, we considered a third category of papers with anomalous peer-review—those that were accepted with remarkably short turnaround times (30 d or fewer between submission and acceptance). Short handling time is regarded by some as a hallmark of paper mill activity (68, 69).† We found a comparatively large number of editors with an anomalous rate of short turnarounds. However, we found a comparatively small number of authors enriched for submissions assigned to editors with anomalous rates of short turnaround times. This suggests that short turnaround may be a less robust marker for identifying potential collusion than retractions or PubPeer comments at PLOS ONE. However, it may still be a valuable indicator of potential editorial misconduct in other contexts.
The 45 PLOS ONE editors whom we were able to flag due to the anomalous rate at which they accepted retracted or PubPeer commented publications or had their submissions handled by other flagged editors comprise only 0.25% of all editors. These individuals edited 1.3% of all articles published in PLOS ONE by 2024 but 30.2% of retracted articles. More than half of these editors (25 of 45) also authored articles retracted by PLOS ONE (SI Appendix, Fig. S1).
To uncover connections between these individuals, we built a network of the publishing relationships among them (within PLOS ONE alone, Fig. 1E). Even though we lack information on relationships that involve other journals, we are still able to find densely connected group of individuals serving as editors between 2020 and 2023 (Bottom Right cluster in Fig. 1E). These editors, affiliated with institutions from four different countries, sent most of their submissions to one another over other editors. More than half of the articles accepted by this group of editors have been retracted with nearly identical notices—“one of a series of submissions for which we have concerns about authorship, competing interests, and peer review” (70).
As a robustness test, we verified that our results were not changed if we used a more strict threshold for significance (SI Appendix, Fig. S2). While we also wanted to test for the impact of potential field-specific rates of retractions, we lacked sufficient statistical power to do so except for cell biology, the field with the most retractions in PLOS ONE. Again, we find our conclusions to be unchanged (SI Appendix, Fig. S3).
These anomalous patterns are not restricted to PLOS ONE. Hindawi journals also disclose the editors of published manuscripts. SI Appendix, Figs. S4–S13 show the results of our analyses for the ten Hindawi journals with the most retracted and PubPeer-commented articles. Across these journals, we find 53 editors who accept eventually retracted articles anomalously frequently (six of whom are flagged for multiple journals), 52 who accept PubPeer-commented articles anomalously frequently (four of whom are flagged for multiple journals), and 96 who accept fast-tracked articles anomalously frequently (seven of whom are flagged for multiple journals).
Similar to how editors handle journal submissions, the articles published through conference proceedings are usually handled by a small team of conference organizers who could also accept an excessive number of problematic articles. We apply the same methodology to all IEEE conference proceedings since 2003 (SI Appendix, Fig. S14), treating each conference proceeding as equivalent to one editor. We flagged 84 conferences for hosting eventually retracted articles anomalously frequently, 158 conferences for hosting PubPeer-commented articles anomalously frequently, and 183 conferences for hosting articles with tortured phrases anomalously frequently.
Consistent with the hypothesis that editorial staff will be largely retained from year to year within a conference series, we find some IEEE conference series for which several installments held on different years are flagged for tortured phrases (SI Appendix, Fig. S15). Among these, we identified at least seven IEEE conference series lasting three or more years where the conference was flagged for anomalously frequent handling of articles with tortured phrases every year the conference has been held.
Coordinated Production of Fraudulent Science.
Anomalous patterns are not only present in the editorial handling of peer-review. Indeed, it is thought that “paper mills” are capable of producing fraudulent research at scale (29, 61, 71, 72). Thus, we investigated whether there is coordination in the production of fraudulent science and in the co-opting of scientific journals by paper mills and whether such coordination leaves a trace in the scientific literature.
To pursue this investigation, we made use of a hallmark of fraudulent science: image duplication. We built a network comprising 2,213 articles flagged for duplicate images (nodes), which are connected through 4,188 observations of image duplication between articles (edges). This network is split among 20 connected components (Fig. 2A). Because some of those components are quite large (maximum size is 622 articles), we used a stochastic block modeling approach for identifying modules within each connected component (73, 74) (Fig. 2B). Despite the fact that image duplication implies that these studies did not occur as described, only 34.1% have been retracted (Fig. 2C).
Fig. 2.
Evidence of coordination in the production of fraudulent science and the targeting of specific journals for its publication. (A) We build a network of articles with comments in PubPeer concerning image duplication. We identify 2,213 articles (nodes) that are connected through 4,188 edges indicative of image duplication across a pair of articles. We identify 20 connected components within this network. (B) Cartographic representation of the network. Each different colored circle represents a module identified using stochastic block modeling (73, 74). The number inside a circle indicates the number of articles in the module. The lines connecting circles indicate that the two modules are part of the same connected component and their width denotes the number of intermodule edges. (C) Fraction of retracted articles for each module. The black wedge denotes the fraction of retracted articles in the module. (D) Distribution of years of publication for articles in each module (Left) and p-value for the null hypothesis that years are distributed randomly across modules (Right). Color indicates year of publication (Left) or log10 of p (Right). (E) Distribution of the publishers of the articles in each module (Left) and p-value for the null hypothesis that publishers are distributed randomly across modules (Right). Color indicates publisher (Left) or log10 of p (Right). Dark saturated red indicates extremely low probabilities (P<10−5), suggesting rejection of the null hypothesis. It is visually apparent for both cases that the null hypothesis is rejected for all modules for which the analysis has sufficient power.
According to our working hypothesis of paper mill operations, paper mills produce and publish articles in large batches. By this model, papers within each batch could use a fixed bank of images (rather than piecemeal appropriation and assembly of images from multiple sources) and the resultant articles would manifest as modules within our image sharing network. Our hypothesis also implies that articles within each module would tend to appear in the same journals around the same time. Thus, one would expect that articles within a module would be heavily concentrated within specific publishers and within specific years, and we should be able to reject the null hypothesis that the distributions of publishers and years of publication obtained for different modules are statistically indistinguishable.
To quantify the degree to which the distribution of years in which the articles belonging to a certain module is anomalously concentrated, we use Shannon’s entropy (Fig. 2D and SI Appendix). We find that for nearly all modules, the publication dates of the articles sharing images are extremely concentrated in time, rejecting the null hypothesis. To quantify the degree to which the distribution of publishers in which the articles belonging to a certain module is anomalously concentrated, we again use Shannon’s entropy. Again, we find that for nearly all modules, the journals publishing articles that share images are extremely concentrated by publisher, rejecting the null hypothesis (Fig. 2E).
Our working hypothesis and these anomalous patterns are consistent with a modus operandi where paper mills cooperate with brokers—or act, themselves, as brokers—who control at least some of the decisions at target journals and can guarantee the simultaneous publication of batches of fraudulent articles in a single journal. The successful implementation of this strategy can be accomplished with the aid of even a small number of defectors at the journal level, as observed earlier (Fig. 1).
Evidence of Journal Targeting and “Journal Hopping”.
An implication of the working hypothesis just discussed is that paper mills have the ability to guarantee publication across sets of journals and publishers. However, over time, certain journals may fall out of favor with a paper mill’s clientele or may otherwise become unavailable. For instance, a journal used by a paper mill may be deindexed by WoS or Scopus, leading to decreased demand for publications in this journal by clientele belonging to academic organizations that only credit articles in indexed journals. Thus, one would expect that the set of journals with which a paper mill operates would tend to change over time. We call this adaptive behavior “journal hopping.”
We were able to uncover an entity that displays this behavior: ARDA advertises “Conferences and Meetings,” “Journal Publication” and “Thesis/Article Writing” on its website. As of June 2024, ARDA’s homepage reported involvement in “4,565+ publications.” Since January 2018, ARDA has listed 188 unique journals on its website as available venues for customers with the list of available journals evolving over time. Of these, 106 journals (56.4%) were indexed by Scopus, 51 (27.1%) were indexed by WoS, 29 (15.5%) were indexed by both, and two (1.1%) were indexed by MEDLINE. Notably, ARDA appears to have significantly expanded its operation, growing from a list of only 14 journals in January 2018 to 86 journals in March 2024 (Fig. 3A). Seventeen (9.0%) of these journals are suspected to be “hijacked journals,” where a journal was once legitimate but a paper mill has gained complete editorial control over the journal and its indexed content (31–33).
Fig. 3.
The business ARDA maintains an evolving portfolio of targeted journals. (A) Number of journals listed on ARDA’s website over time. We identify 188 unique journals listed by ARDA. Since 2021, ARDA has maintained a portfolio of 70+ journals in which they guarantee article publication. (B) Estimated time span over which individual journals were listed by ARDA. We represent each journal as a horizontal line and use color to indicate source of journal indexing (WoS, Scopus, or both). We start a bar on the earliest snapshot date we find it listed on ARDA’s website and end it on the final snapshot date. We show year of deindexing by an empty circle at the same level as the journal bar, placing it at the start of the year (due to a lack more detailed information on exact timing), followed by a dotted line from this point to the last time the journal was listed on ARDA’s website. Note that deindexing can be retroactive (e.g. Scopus could, in 2020, deindex a journal with an effective date in 2018).
We find that the probability that a journal listed by ARDA is deindexed far exceeds the baseline rate—13 out of 39 (33.3%) Scopus-indexed journals listed by ARDA in 2020 were later deindexed, versus 147 out of 27,197 (0.5%) Scopus-indexed journals actively publishing in 2020 (two-tailed Z test of proportions, P<0.0001). The evolution of ARDA’s portfolio of journals often appears to occur in direct response to deindexing. For instance, a group of journals deindexed by Scopus in 2020 or 2021 were all removed from ARDA’s website in May 2021 and subsequently replaced with new journals. To our knowledge, this is the first reported case of an entity engaging in fraudulent publishing that itself engages in journal hopping (Fig. 3B). The rapid rise in annual publication volume of some journals apparently targeted by these entities (SI Appendix, Fig. S16) is consistent with the hypothesis that journal hopping is a widespread practice.
Many articles published in the journals listed by ARDA are well outside the journal’s stated scope (e.g. an article about roasting hazelnuts in a journal about HIV/AIDS care or an article about malware detection in a journal about special education). For the set of five journals that we inspected comprehensively, we found that between 34.0% and 98.7% of the articles published in these journals were outside of the journal’s stated scope (Datasets S7 and S8).
Among these journals, we also found many publications with authors from multiple countries (10.1%), supporting the hypothesis that paper mills will sell authorship slots on individual manuscripts but contrasting with the hypothesis that paper mill products might be recognizable by the lack of international collaboration in their authorship lists (30). Paper mills, predatory journals, and brokers likely operate under a number of author procurement models (75), including models where local scholars are targeted (resulting in a lack of international collaboration in authorship lists) (72) and models where authorship slots are sold to scholars worldwide (resulting in implausible international collaborations) (61).
Differential Prevalence of Fraud Within Disciplinary Subfields.
Our results show that networks of individuals and entities act to produce fraudulent manuscripts, to select journals and publishers for targeting, and to facilitate their publication in journals indexed by aggregators such as WoS and Scopus.
Next, we investigated whether certain subfields are preferentially selected by those involved in scientific fraud. We restricted our analysis to closely related and similarly sized subfields in the biology of RNA that have each seen recent increases in popularity. We further restrict our attention to six subfields of interest to RNA biologists, namely CRISPR-Cas9, transfer RNAS (tRNAs) and development, tRNAs and cancer, circular RNAs, micro-RNAs (miRNAs) and development, miRNAs and cancer, and long noncoding RNAs (lncRNAs), and download bibliometric information on articles returned when searching in PubMed (exact search strings shown as titles in Fig. 4). Among these closely related subfields, paper mills are suspected to be particularly drawn to miRNAs, circular RNAs, and lncRNAs (76–78).
Fig. 4.
Closely related subfields of biomedical research display similar rates of errata, but different rates of retractions. We investigated rates of retraction and errata for six subfields within RNA biology, focusing mostly on subfields that had their onset since 2000 and have experienced a dramatic increase in the number of annual publications (Left column). The dotted lines show all publications, including reviews, while the full lines exclude reviews from the count. While annual number of publications or their growth rate can differ significantly, all subfields have similar rates of errata. In contrast, the rate of retractions shows remarkable differences across subfields (Right column). Most rates decrease after 2019, presumably due to time censoring (i.e., not all retractions or errata may have yet occurred or been reported). Studies concerning “lncRNAs” and “miRNAs and cancer” show retraction rates which peak at around 4%, higher than even the rate at which errata are written. These rates are unlikely to be compatible with good-faith mistakes and point toward concerted attempts at scientific fraud.
For each of these six subfields, we calculate the annual number of publications and the annual rates of errata and retractions for the period 2002–2022 (Fig. 4). We calculate the publication rate of errata because it provides a control or baseline for understanding the rate of retractions. For all subfields, we find that the publication rate of errata hovers between 1.5% and 2.5%. This suggests that, within each subfield, the extant literature is revisited at relatively similar rates.
As we surmised, we do not find the same consistency for the rate of retractions. Consistent with most scientists’ expectations concerning egregious errors or scientific fraud, for CRISPR-Cas9, we find that the rate of retractions is only about 0.1%. The rate of retractions increases from tRNA (peak of ∼1% for tRNA and cancer) to circular RNAs (peak of ∼2.5%), miRNAs (peak of ∼4% for miRNA and cancer), and lncRNAs (peak of ∼4%). We also find higher rates of retractions for subfields focused on cancer than on development.
Retraction rates can be highly variable across publishers for articles in these subfields. Indeed, for studies concerning “lncRNAs” and “miRNAs and cancer” published in certain journals (SI Appendix), the retraction rate exceeds 10%, while for some other journals the rate is close to zero, far below expectation (SI Appendix, Fig. S17).
Scientific Fraud Is Growing Much Faster than the Scientific Enterprise As a Whole.
Several studies have recently attempted to characterize the scale of published paper mill products in relation to the scale of the overall scientific literature (30, 79). Acceptance of those estimates has been hindered by limitations in the field’s ability to unambiguously recognize articles produced by paper mills, by the heterogeneous rates of fraud by discipline (Fig. 4) and by the difficulty in conceiving that the enterprise of scientific fraud is sufficiently large or coordinated.
Speaking to the degree of coordination and scale of the entities involved in scientific fraud, we find that retraction notices by journals are now published mainly in batches of more than 10 articles (Fig. 5A). One would expect retraction notices published simultaneously to be for related reasons. Indeed, retraction notices released by PLOS ONE are consistent with this expectation (SI Appendix, Fig. S18).
Fig. 5.
Articles of fraudulent provenance have an apparent growth rate greater than that of the entire scientific enterprise and already far outpace the scope of science integrity measures currently in use. (A) Retractions are increasingly published in batches. The ∼2010 spike in the number of large-batch retractions is almost entirely attributable to a large swath of conference proceedings articles retracted by IEEE. For the first time since this spike, the majority of 2023 retractions were reported in batches larger than 10 articles. (B) Annual global scientific activity as measured by items labeled as “journal article” or “conference proceeding article” in OpenAlex (47), as retracted articles reported by Retraction Watch, as PubPeer-commented articles and as suspected paper mill products. We make use of the linear trends observable in the log–linear plot to extrapolate these observations for the period 2020–2030. We show the 95% CI using shaded bands. The number of suspected paper mill products shows the largest growth rate, with a doubling time of 1.5 y. (C) Annual global scientific activity captured by WoS as measured by the number of actively publishing journals, the number of journals deindexed annually by WoS, the number of journals with retractions, the number of journals with PubPeer comments, and the number of journals with suspected paper mill products. It is visually apparent that deindexing now occurs at a level far below the level of occurrence of journals publishing suspected paper mill products. These patterns hold for Scopus and MEDLINE (SI Appendix, Figs. S21–S23).
We also find that the number of retracted articles has been increasing exponentially over the last 30 y (Fig. 5B). Remarkably, and testifying to the enormous impact of postpublication review efforts, we find the number of articles with PubPeer comments has also been increasing exponentially. To provide perspective, we note that the number of retracted articles and PubPeer-commented articles has been doubling every 3.3 y and every 3.6 y, respectively, while the total number of publications has been doubling every 15.0 y (80, 81). However, suspected paper mill products have been doubling every 1.5 y (SI Appendix, Figs. S19 and S20). Notably, suspected paper mill products now outnumber annually retracted articles and are projected to soon outnumber the number of PubPeer-commented articles.
As discussed earlier, journal deindexing is a powerful mechanism available to those defending the integrity of the scientific literature. Bibliometric aggregators can index tens of thousands of actively publishing journals. In response to concerns about editorial practices, some of these aggregators can deindex a journal. WoS and Scopus deindex on the order of a hundred journals each annually. While this may appear to be a large number, it is ten-fold smaller than the number of journals that publish paper mill products (Fig. 5C and SI Appendix, Fig. S21).
Limitations
A limitation of our study is the comprehensiveness of the data we consider. Our analyses rely on the instances of scientific fraud that have been reported. It is likely that many fields and journals are underrepresented in the corpora we consider. Indeed, the consensus among experts is that the vast majority of paper mill products have not been detected (30, 79, 82). Further, some of our case studies focus on particular disciplines, outside of which our findings may not be generalizable.
Additionally, temporal changes in detection effort or in the attention paid to different fields may produce spurious trends. Indeed, the many unknowns about the global enterprise of scientific fraud leave open the possibility that the scale of systematic fraudulent activity has always been large but that only now has been detected. We comment further on this possibility in SI Appendix.
Discussion
Competition for limited funding and jobs pushes scientists and the organizations that employ them to continually strive toward increasing scale, efficiency, impact, and the growth of the metrics by which these are evaluated. While a select group of nations, organizations, and individuals at the pinnacle of the scientific enterprise have access to the resources necessary for such extraordinary growth, most do not. Research suggests that perceived injustice is associated with research misbehavior (16) and that lack of opportunity and training are often cited as drivers of misconduct (27, 83). Thus, increasing inequality in resource accessibility may be contributing to the increasing scale of scientific fraud. However, even when provided with resources, research remains a high-risk activity—one does not know a priori whether a study will be successful or not. Why risk failure, jeopardizing one’s career, when for a relatively small fee one can easily acquire the publications and citations that would otherwise require an immense amount of labor?
Discussions with different stakeholders suggest that many currently perceive