BEST AVAILABLE SCIENCE: METRICS FOR EVALUATION OF SCIENTIFIC CLAIMS

For several decades, the staff of the Institute for Regulatory Science has been involved in the development of a process to evaluate the validity of scientific claims. We chose the term Best Available Science (BAS) to generically sort out the validity of claims and counter claims. These efforts led to the development of Metrics for Evaluation of Scientific Claims (MESC). Our initial review of literature could not find the term “best available science”, much less a definition for it. Ultimately, our choice was based on the following:

1. Best connotes scientific information with the greatest degree of excellence and authenticity based on sound logic, which includes good judgment, reasoning, or evidence. It indicates the highest level of reliability. However, it suggests that there is also scientific information that does not meet the highest level of reliability. It implies that on occasion a specific area of science has not reached a reasonable level of maturity. There is an implication that one has to generically identify the level of reliability of that specific area of scientific information. Consequently, it implies the need for some sort of standard.
2. Available connotes scientific information that is accessible and attainable, i.e., capable of being obtained or used, based on No. 1 above.
3. Science excludes any subject that is not science.

During the evolution of BAS/MESC we recognized the need to identify basic principles that must be used in assessing scientific claims. We identified four principles that historically have led the scientific community in virtually all new findings:

Open-mindedness Principle implies the willingness to accept new knowledge.

Skepticism Principle requires that those who make a scientific claim must provide sufficient evidence supporting their claim.

Universal Scientific Principles imply that there are certain principles applicable to virtually all scientific disciplines.

Reproducibility Principle implies that any qualified investigator should be able to reproduce the claim.

The BAS concept and MESC derived from it are based on three Pillars as follows:

1. Standardization of scientific information
2. Categorization of reliability of scientific information
3. Areas outside the purview of science

PILLAR: STANDARDIZATION OF SCIENTIFIC INFORMATION

Scientific information can be generically classified into the following classes:

Class I – Proven Science

This class consists of scientific laws sometimes called scientific principles and all other information that have been unequivocally confirmed and are generally accepted. The cornerstone of this class is that any investigator who has the proper equipment and the necessary skills can reproduce it. Similarly, this class of scientific information does not require assumptions or any other prerequisite for its proof. This class also includes the application of scientific laws to various branches of commerce and industry— provided the laws are correctly interpreted.

Class II – Evolving Science

The overwhelming scientific advances in virtually all disciplines are evolving science.

Reproducible Evolving Science: Reliable information dealing with a subject that is not completely understood constitutes the core of this class. The key factor in placing information into this class is reproducibility. Advancements in virtually all branches of science are based on the desire of investigators to improve knowledge in this class.

Partially Reproducible Science: Formerly referred to as Rationalized Science or Scientific Extrapolation, this class includes certain segments of regulatory science information including predictive models. Although it builds upon Proven or Reproducible Evolving Science, it uses assumptions, extrapolations, and default data to derive its results. An important characteristic of this class is its level of reproducibility. Whereas the scientific foundation of this class meets the Reproducibility Principle, the choice of assumptions, mathematical processes, default data, and numerous other prerequisites are inherently arbitrary and thus are not necessarily reproducible.

Evidence-Based Science: This class attempts to correlate systematic observations performed in accordance with Universal Scientific Principles to an effect. There is extensive literature covering this class including a large segment of epidemiology. Experience shows that correlation does not necessarily imply causation and as expected, some correlations have correctly identified their cause but others have proven to be unrelated. Much of evidence-based medicine belongs to this class.

Hypothesized Science: This class consists of an organized response to an observation, an idea, or any other initiating thought process. Imagine that an event is observed by a scientist who cannot explain it by any known scientific principles, approaches, or other processes. Subsequently, the scientist hypothesizes one or more potential causes. Whereas some Hypothesized Science has proven to be worth pursuing, often the conclusions are proven to be wrong.

Scientific Judgment: On occasion, decisions must be made without having the needed scientific information including basic principles, the necessary data, and other scientific requirements. The methodology for expert judgment is reasonably well-developed, and consists of asking a number of presumably knowledgeable individuals to give answers to specific questions and statistically assess the results. However, this class is often an educated guess.

Speculation: This class consists of information that cannot meet standards described in any of the above classes. It is often based on the intuition of an individual who wants to stimulate a discussion or initiate a research project.

Fallacious Information

Information that clearly falls into the purview of science but is inconsistent with the three classes identified above is fallacious information. This information is often called “pseudo science”, “junk science”, or “politically-processed science”.

PILLAR: CATEGORIZATION OF RELIABILITY OF SCIENTIFIC INFORMATION

There are rational and reasonable uncontested methods to resolve scientific controversies. Briefly, scientific information is divided into the following four distinct categories:

Category 1 - Personal Opinions: Expression of views by any individual is included in this group. At best, this category can be used to initiate the study of a scientific issue.

Category 2 - Gray Literature: Written information that has not been subjected to an independent peer review is included in this category. This is the favorite category of government agencies, advocacy groups, and individuals who want to promote an idea. Experience shows that in the overwhelming majority of cases, this category does not meet the requirements of scientific acceptability. At best, this category should be used to initiate a study.

Category 3 - Peer‑Reviewed Science: Information subjected to an independent peer review constitutes this category. Peer review is the foundation of scientific acceptability. There are numerous requirements for acceptability of peer review. Briefly, the individual who is chosen as a reviewer must be a “peer” to the author of the study, and must have no conflict of interest. In addition, the author of the study must respond to criticism by the peer to the satisfaction of an uninvolved person or organization.

Category 4 - Consensus-Processed Science: This category consists of information resulting from a process used to resolve scientific disputes. The prerequisite for this process is the formation of a group of peers under the auspices of an organization that is uniquely qualified to do so. Category 4 is a formal process requiring consideration of qualifications of the members of the consensus panel and their potential conflict of interest, which is contrary to the view that is often expressed.

PILLAR: AREAS OUTSIDE THE PURVIEW OF SCIENCE

The concept of BAS deals with classification of scientific information and evaluation of scientific assertions. It specifically excludes areas that are outside the purview of science. The inclusion of faith, ideology, beliefs, or any other non-scientific objectives in assessing the validity of scientific information is inconsistent with the foundation of BAS or MESC.

About RSI

HOME
OUR MISSION
CONTACT US
CENTERS

Science at RSI

BAS/MESC
What is Regulatory Science
Studies in Progress

Centers

TIMC
ICESH
RAC
SESC
STARC

Publications

Regulatory Science and Technology
Manuals
Independent Peer Reviews
Independent Scientific Assessments
Stakeholder Participation Reports