Accepting Goodhart’s Law in Academia

https://www.youtube.com/watch?v=D9Qf4Ft-1nQ&list=PL3C6eF-zu5AYohNL1ZgOBqlwwJ29x-lTO&index=13

What is Goodhart's Law?

• The law states, "When measurement becomes a goal, it ceases to function as a true indicator. When researchers pursue metrics such as "number of papers" and "number of citations" too much, they are more likely to focus on quantity rather than quality, and they are more likely to engage in fraud.
• Related:.Incorrect KPI setting

Waste of research funds

• It refers to the current situation in which the majority of papers lack reproducibility (more than 50% of studies cannot be reproduced) and numerous research funds are spent on "unreliable results". Not only do they fail to produce results, but negative results are never published in the first place, resulting in a serious waste of funds.

Publication and Quality of Science

• Conventional evaluation axes such as the number of citations and impact factor are difficult to measure the essential quality of research. Some researchers tend to give priority to the number and impact of papers rather than the content, aiming for "high impact journals," or to report only favorable results.

Structure that is ridiculous / only a few are accepted

• There is a tendency for researchers to think that they are not recognized as researchers unless their work is published in a prestigious journal, or that they are not highly regarded if they do not submit many papers, which tends to downplay research that emphasizes originality and careful, time-consuming research.

H-index

• This is a representative indicator of researcher productivity. It is a combination of the number of papers published and the number of citations, but it also encourages behavior focused on "raising the number" (mass production of papers, self-citations, etc.) and tends to diverge from the original evaluation of quality.

Problem of not sharing negative results

• Although sharing negative (contrary to one's intentions) results is important to promote reproducible research, sharing is discouraged because it is difficult to be cited and does not lead to career opportunities. As a result, reproducibility decreases and research efficiency declines.

Peter Higgs example

• Peter Higgs, known for the Higgs boson, was the type to make a big impact every few years, but he himself stated that it is difficult to be valued in the current competitive environment where "frequent papers are a must.

Reddit for Science（ResearchHub）

• A place like Reddit where research results and papers are shared and tokens are awarded in exchange for upvotes. Unique tokens ($RSC) are rewarded for submissions and peer review (peer review).
• ResearchHub has introduced a design that offers bonuses for actions that improve the quality of research, such as pre-registration (pre-registration) and open data. For example, pre-registration papers receive double the rewards.
• Voting by DAO
  • The community (DAO) will vote to determine how much to reward what kind of research behavior. The management does not unilaterally set the rules, but the token holders discuss and reach a consensus on what kind of research is desirable.
• $RSC Rewards
  • Tokens awarded for submitting papers, peer review, and contributing to open access and data publication. A mechanism that allows researchers to gain financial benefits by earning these.

Talk about people not interested in token rewards.

• Researchers who are already well-established and those who do not necessarily need monetary rewards may be reluctant to accept token rewards. However, for young researchers and post-doctoral researchers, it could work as a significant incentive to open up their research.
• In fact, many people participate because they share the philosophy of "open scientific improvement" rather than tokens. The community value of voting for each other in DAOs also adds to this, and in some cases, motivates people to participate in addition to the rewards.

summary In academia, where Goodhart's Law has made traditional citation-based metrics an objective, to the detriment of research quality, ResearchHub is attempting to "set the ideal research behavior" through token rewards and DAO voting. Specifically, the system encourages behaviors that increase reproducibility and transparency, such as open access, prior registration, and open data, and awards tokens ($RSC). In this way, the project is attempting to create a new type of research incentive system, involving young researchers and the research community at large.

This page is auto-translated from [/nishio/Accepting Goodhart’s Law in Academia](https://scrapbox.io/nishio/Accepting Goodhart’s Law in Academia) using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I'm very happy to spread my thought to non-Japanese readers.