Materials and Methods

For the purposes of the present report, a study was defined as a report that included Materials and Methods and Results sections. A study could describe more than 1 antimicrobial-to-control comparison. A manuscript was defined as a publication containing reports of 1 or more studies. A technical bulletin from a pharmaceutical company reporting a study was also considered a manuscript.

A search of the electronic citation databases AGRICOLA, Commonwealth Agricultural Bureau, and PubMed was conducted to identify relevant manuscripts published between 1970 and 2005. These electronic searches included terms for the population, disease, and treatment of interest. The full search string was [beef, bovine, calf, calves, cattle, cow(s), dairy, Hereford, Holstein, ruminant(s), steer(s)] AND [bovine respiratory disease, bovine viral diarrhea, bovine viral diarrhea virus, undifferentiated fever, BRD, BVD, BVDV, Haemophilus somnus, Histophilus somni, IBR, infectious bovine rhinotracheitis, Mannheimia hemolytica, Pasteurella multocida, pasteurellosis, respiratory disease, undifferentiated bovine respiratory disease] AND [amoxicillin, ampicillin, antibiotic(s), antimicrobial(s), erythromycin, ceftiofur, cloxacillin, enrofloxacin, flor-fenicol, gentamicin, lincomycin, oxytetracycline, penicillin, spectinomycin, sulfamethoxazole, tilmicosin, trimethoprim, tylosin]. Further, the reference lists of relevant manuscripts and the table of contents of the Proceedings of the American Association of Bovine Practitioners and the World Buiatrics Association from 1990 to 2005 were hand searched to identify any additional manuscripts. We did not include years prior to 1990 because we expected articles that had appeared in these proceedings prior to 1990 had had sufficient time to be submitted for publication. An additional database associated with the Veterinary Antimicrobial Decision System was also searched. In addition, via a letter addressed to technical services representatives, companies marketing antimicrobials with a label claim for treatment of BRD were invited to provide details of any unpublished studies. The websites for companies marketing antimicrobials with a label claim for treatment of BRD were also searched to identify any relevant clinical trials.

Identified manuscripts were included in the study if they described at least 1 clinical trial evaluating a control and 1 or more comparison groups, had been conducted at a feedlot in North America, and involved cattle with naturally occurring BRD, as defined by the researchers conducting the trial. Two independent reviewers (the first author [AMO] and 1 of 3 veterinary students [LF, NGW, or MR] completing a summer internship program during 2005 to 2008) read the abstracts of manuscripts identified by the search, and if both reviewers deemed the manuscript irrelevant, the manuscript was not considered further. Full copies of manuscripts considered potentially relevant following this initial screening were obtained, and manuscripts were again assessed for relevance on the basis of the full manuscript. Manuscripts that were still considered relevant following this evaluation were included in the study.

For manuscripts included in the study, data regarding key study design features were extracted by a single reviewer and independently verified by a second reviewer. Information obtained included the source of funding and whether authors were clearly identified as employees of a pharmaceutical company. It was assumed for company technical reports that the company funded the study and employed the authors. For each manuscript, the number of studies reported and whether the author included a description of the following were determined: enrollment criteria, randomization procedures, blinding procedures, number of withdrawals, primary outcome for determination of sample size, sample size justification, null hypothesis, and number of outcomes for each individual study. The rationale for considering each of these items as a key study design feature has been described in detail previously.⁵ Briefly, enrollment criteria should be reported so that readers can assess the potential for selection bias at enrollment and can evaluate the external validity of the study population, randomization procedures should be reported so that readers can assess the potential for confounding bias, and blinding procedures should be reported so that readers can assess the potential for allocation and information bias. The remaining 5 key study design features (ie, number of withdrawals, primary outcome for determination of sample size, sample size justification, null hypothesis, and number of outcomes reported) were all included to understand the potential for analytic bias.

To investigate the association between reporting of key study design features and a favorable study outcome, a metaregression approach was used. In general, in evaluations of antimicrobial treatments, 2 trial approaches may be used. The trial may be designed to assess whether treatment A is superior to treatment B (ie, a superiority trial), in which case the null hypothesis should be that treatment A is equal to treatment B. Alternatively, the trial may be designed to assess whether 2 treatments are equal (ie, an equivalence trial), in which case the null hypothesis should be that treatment A is not equal to treatment B. For either trial approach, a favorable outcome is defined as rejection of the null hypothesis.

Therefore, the null hypothesis for the present investigation was that studies that failed to report key study design features were equally likely to reject the null hypothesis for the primary outcome as were studies that did report these key study design features. Because studies may report multiple outcomes, we only tested this hypothesis for the primary outcome of interest, which was defined as the outcome for which the sample size was justified.⁵

Results

The literature search yielded 809 manuscripts, of which 44 were considered relevant by at least 1 reviewer on the basis of the abstract and title. After evaluation of the full manuscripts, 15 of these 44 manuscripts were excluded as they did not meet the relevance criteria. The remaining 29 manuscripts,^6-33,a which described 41 studies, were considered relevant and were included in the study. One manuscript described 4 studies,⁶ 3 manuscripts described 3 studies each,^7,8,28 and 1 manuscript described 2 studies.⁹ Twelve of the 29 (41%) manuscripts did not disclose a funding source; however, 21 of the 29 (72%) manuscripts identified an author as an employee of a pharmaceutical company.

At the study level, 36 of the 41 (88%) studies reported a random method of treatment allocation. Treatment allocation was at the individual animal level for all 41 studies. Of the 36 studies that reported random allocation, 9 provided a description of the method of allocation sequence generation, as has been recommended.⁵ For some studies, it seemed possible that a nonrandom process was described as random, such as “calves were ranked from highest to lowest weight, and randomly assigned alternatively to either the florfenicol or the tilmicosin group.”¹⁰

Twenty of the 41 (49%) studies reported that study investigators were blinded to treatment group. Five of the 41 (12%) studies failed to report the use of both randomization and blinding; however, 20 of the 41 (49%) studies reported the use of both study design features. Descriptions of blinding were generally poor, in that the method of blinding was not described or it was unclear who was blinded (ie, treatment allocators, animal caregivers, or outcome assessors).

Eligibility requirements and withdrawals were frequently described. Thirty-eight of the 41 (93%) studies reported eligibility requirements for enrolled animals, and 39 of the 41 (95%) discussed what would constitute an animal's withdrawal from the study.

Only 3 of 41 (7%) studies included a rationale for the size of the treatment groups; therefore, it was possible to discern the primary outcome for these 3 studies. No studies described the null hypothesis to be tested. Thirty-seven (90%) of the studies had at least 3 outcomes, 4 (10%) studies reported 2 outcomes, 18 (44%) studies reported between 3 and 5 outcomes, and 18 (44%) studies reported between 5 and 10 outcomes. One study reported 14 outcomes, the largest number of outcomes reported.

Metaregression analysis was not conducted as proposed because it was not possible to assess whether studies that failed to report key study design features were more likely to reject the null hypothesis for the primary outcome than were studies that did report these key study design features since it was not possible to discern the primary outcome for most of the studies. Further, because the null hypothesis was not stated for any studies and all studies reported multiple outcomes, it was not possible to discern whether the purpose of these studies was to assess the equivalence or superiority of the treatments; therefore, it was not possible to determine the direction of a favorable bias.

Discussion

Results of the present systematic review suggested that many studies of antimicrobial treatment of BRD in North American feedlots did not report key study design features that would assist critical evaluation by readers. We recommend that study designers, authors, reviewers, and editors request this information when it is lacking. Such reporting will lead to a better understanding of the importance of these features in feedlot studies and will allow readers to assess the study features used, rather than rely on assumptions when this information is missing from study reports.

It was promising that many studies identified in the present review described the criteria used to identify an animal's eligibility for the trial (38/41 [93%]), whether animals were removed or withdrawn from the study (39/41 [95%]), and whether randomization (36/41 [88%]) or blinding (20/41 [49%]) had been used. However, the failure of all studies to provide sufficient information for readers to assess the validity of the results because the null hypothesis was not stated, the study size was not justified, or multiple outcomes were reported was of considerable concern.

The present review assessed the reports of trials rather than the execution of trials, so it was not clear whether the studies failed to use the specified design features or omitted reporting them. Previously, it has been documented that failure to report key study design features such as randomization and blinding is associated with more positive outcomes in human medicine^1,2 and in 1 study³ in veterinary medicine. It was encouraging that many studies used these features. Further, it seems that some of the studies that did not report randomization may have used this tool. For example, 1 author of a study¹¹ in which blinding and randomization were not reported was also a coauthor on 2 other studies^12,13 that did use these design features. Another study included a coauthor who had previously published a report¹⁴ describing the importance of blinding and random allocation in vaccine trials in feedlots. It was possible that in these instances, reporting omissions were more likely than design omissions. However, there is still progress to be made in the description of randomization. Random has a very precise definition,⁵ and some studies described processes as random that may not indeed be random. When the study setting necessitates that a nonrandom process of allocation be used, researchers should describe the approach to allocation and allow the reader to assess the potential for bias, rather than not describe the allocation method.

With respect to blinding, the same arguments can be made. This feature was used reasonably commonly; however, the descriptions were inadequate. How blinding was achieved should be described for 3 phases of the study—at allocation to treatment groups, during care-giving, and during outcomes assessment—because each of these phases has the potential to introduce bias.⁵ Occasionally, blinding is not possible, but when blinding is not possible, researchers should report transparently why blinding was not used. It should not be assumed that objective outcomes such as weight gain were unbiased because differential caregiving could have still introduced a bias in objective outcomes.

Several of the study design features evaluated in the present review were not design execution features, such as randomization or blinding, but were instead plan-ning-related features. For example, determination of the null hypothesis, primary outcome, and sample size would all occur before the study was executed, and the results of our review indicate that it has not previously been standard to include these features in trial reports. However, it is clear that the reporting of these features in the Materials and Methods of a trial report can add substantially to clarifying the study objective and facilitating the interpretation of the study outcome. In the future, we suggest that strong consideration should be given to including this information.

The time and expense of field trials warrant detailed examination of the treatment tested; however, the conduct and analysis of trials must be planned prior to the study to avoid the introduction of bias into the study outcome.³⁴ The best way to communicate this careful planning is to clearly state the objective of the study and the null hypothesis tested to achieve the objective.⁵ It follows that once the null hypothesis has been stated, it will be clear what the primary outcome will be, the magnitude of the difference in the primary outcome that the authors believe is clinically relevant, and the sample size required to detect such a difference. All of these features were lacking from most studies examined in the present review. The rationales for these features have been described in detail.^5,34,35

Although many of the 41 studies in the present review described an objective, none provided a null hypothesis. Objectives and hypotheses may appear similar; however, both are needed to assess the validity of a study.⁵ An objective describes the question the study is designed to answer and often includes general terms that can be assessed by a variety of outcomes. For example, the objective of a study may be to test the efficacy of several dosages. In contrast, hypotheses are statements that can be directly answered by use of statistical methods.⁵ A clear statement of the null hypothesis must be included because this allows readers to identify the implications of significant findings or the lack thereof. In studies reporting antimicrobial treatment for BRD, knowledge of the null hypothesis is particularly important to determine whether the study is a superiority study or an equivalence (noninferiority) study and to assess the validity of the sample size and statistical methods used. For superiority studies, the null hypothesis should state that the treatments were equal in effect, and the alternative hypothesis should state that the treatments differed in some way. However, some studies included in the present review reached conclusions that suggested an equivalence study had been performed (ie, statements such as “the results demonstrate the equivalency of both treatment regimes”¹⁵ or “were as clinically effective”¹⁶). For studies to assess equivalence of 2 treatments, the null hypothesis should be that the treatments are different, and rejection of the null hypothesis enables the researcher to conclude that the treatments are equivalent.³⁶ It is inappropriate to conclude that treatments are equivalent if the statistical test used was premised on a null hypotheses of no difference. Equivalence studies require different sample sizes and statistical tests than do superiority studies.^36,37 Unfortunately, because of the lack of reporting of the null hypothesis in the studies included in the present review, it was not possible to evaluate whether the statistical analyses were correctly conducted or the sample size was adequate for the stated purpose or objective. It would substantially improve the available published literature if authors made it clear whether the purpose of the study was superiority or equivalence and specifically stated the null hypothesis being tested. Because of the failure to report the null hypothesis, we could not evaluate the association of study design features with reporting of a favorable outcome in the present review because we could not identify the direction of a favorable bias in many of the studies that were evaluated.

The failure to report the null hypothesis also meant that we could not assess the validity of the sample sizes used in the studies evaluated in the present review because neither the primary outcome used to determine the sample size nor the difference in treatments for superiority trials (or acceptable difference for equivalence trials) was clear. The outcome for determination of the sample size is referred to as the primary outcome.⁵ Only 3 studies^11–13 reported the sample size, and all shared some of the same authors. An adequate sample size provides the study with a basis for yielding statistically significant information, regardless of whether the study leads to rejection or acceptance of the null hypothesis. Researchers should provide a basis for identifying statistically significant findings, as this will allow the reader to assess the role that chance may have played in the study outcome. However, the failure to provide sample size calculations was further complicated by the common use of multiple outcomes in the studies included in the present review. When designing a test of statistical significance, researchers should specify the level of type I and II errors that is acceptable. However, should a researcher design a test without a primary hypothesis and with the intention to carry out significance tests for multiple outcomes, the risk of errors substantially increases.¹⁵ The CONSORT statement⁵ was developed to alleviate the problems arising from inadequate reporting of randomized controlled trials and recommends that researchers limit the number of outcomes to 1 or 2 so that the study-wise error is minimized.⁵ Furthermore, it is recommended that authors clearly state the sample size rationale and indicate any post hoc analyses that are performed. Reporting this information ensures that the reader is aware of the role that chance may have played in the study results. In the present review, 37 of 41 studies reported 3 to 14 outcomes, raising the concern that significant findings were incorrectly reported.

The present review was limited to studies reporting field trials assessing antimicrobial treatment of BRD that were published between 1970 and 2005. This constituted a 35-year period for research in the area of BRD treatment. It is unlikely that we identified all of the studies performed; however, the extensive search suggested that the studies included were a reasonable representation of the available literature. Importantly, the issue of publication bias was not addressed by this review, and many other studies were probably conducted and not reported. However, our objective related to the quality of reporting of published studies, and the search results were likely representative of this group of publications. Objection may be made that we did not attempt to contact any of the authors to determine whether key study design features were used and not reported. Our rationale for not contacting authors was that this additional information would not reflect the aim of the present study, which was evaluation of published information available for use by readers.

More comprehensive reporting of study design features will better assist readers in the critical assessment of feedlot and similar studies. We would recommend that authors, reviewers, and editors request the reporting of this information.