Evaluating studies: First weigh all the evidence

Evaluating studies: First weigh all the evidence

Dec 01, 2008

All of us, including myself, are guilty of skimming journals and reading the abstract to get to the bottom line. We erroneously assume that there is no need to read through the details of the study and that the review process will eliminate any flaws in a piece of research. This assumption is flawed too. Why? One study evaluating human medical journals found that 76 percent of the studies in a well-recognized journal were flawed.

Most veterinarians have not been trained to be critical of published manuscripts. Too often we take the information from a source at face value. This approach hinders our effort to elevate the status of veterinary medicine and the quality of medicine we practice.

There is a current emphasis in the veterinary field to practice evidence-based medicine (EBM). The concept of EBM is simplistic, although the practice is not always easy.

EBM is the practice of clinical medicine based on evidence that has been proven. Sources of this proven evidence include journals, abstracts, textbooks, conferences and other types of publications.

Unfortunately, the evidence in veterinary medicine comes from poorer sources compared to that in human medicine. In EBM, the quality of medicine being practiced is directly related to the accuracy of the information provided to the clinician.

Veterinarians need to be able to critically evaluate these sources to ensure that the conclusions are supported by the evidence.

We have several different sources of information. Primary sources would include publications that contain original research, which is the focus of this article. Secondary sources of information would include literature reviews, professional magazines and lectures. Tertiary sources of information include textbooks and clinic manuals.

The accuracy of the information from these sources is only as good as the primary sources. In both secondary and tertiary sources, the data can be biased, based on the opinion of the author(s).

It is important to make the distinction between manuscripts that are published in peer-reviewed journals and those that are not. Articles in peer-reviewed journals are considered superior. In a peer-reviewed journal, an editorial board made up of specialists in the field reviews and edits the articles submitted and then decides which ones are acceptable for publication.

Keep in mind that even articles published in peer-reviewed journals can be flawed. It is possible that an article is reviewed by a committee that does not have expertise in the field of the paper and relies on other factors to determine the worth of the paper, including the reputation of the authors and institutions or the estimated importance or relevance of a particular article rather than the actual science.

Most journal articles are broken into several sections, including an abstract, introduction, materials and methods, discussion, conclusions and references.

The abstract provides a summary of the work and includes the study design, analysis, results and conclusion. The introduction should review what is known on the topic and provide the rationale for the study. It should also outline the hypothesis and objectives.

The materials and methods section should clearly lay out the design of the study, including what type of study it is, inclusion and exclusion criteria, how a control group was selected, what treatment protocol(s) were used, what variables are being evaluated, how outcomes were measured and how the data will be analyzed. In short, it should allow another investigator to duplicate the study.

The results section should present the data and the results of statistical analysis. There should be no attempt to interpret the data. In theory, enough data should be present to allow the reader to determine the accuracy of conclusions.

In the discussion and conclusions, the authors should state whether the hypothesis was proven and the significance of it. The authors should discuss the strengths and limitations of the study and suggest future applications or studies based on their data.

The final parts are the references and acknowledgements.

Retrospective studies

Table 1: Types of studies
Table 1 (p. 7S) lists the different types of studies. Studies can be classified either as retrospective or prospective.