• View in gallery

    Percentage of Holstein calves inoculated with Mycoplasma bovis that had pulmonary consolidation scores at various cutpoints as determined at necropsy (n = 154). Pulmonary consolidation for each lung lobe was assigned a percentage by a veterinary pathologist. Total pulmonary consolidation scores were calculated on the basis of a reported formula.10

  • View in gallery

    Box-and-whisker plots of estimated sensitivity values for CISs assessed in calves (n = 154) by 8 observers at each pulmonary consolidation score cutoff. Boxes represent the 25th and 75th quartiles, and the horizontal line within the boxes represents the median. Whiskers represent the minimum and maximum values. See Figure 1 for remainder of key.

  • View in gallery

    Box-and-whisker plots of estimated specificity values for CISs assessed by the observers represented in Figure 2. See Figures 1 and 2 for remainder of key.

  • View in gallery

    Graph of median sensitivity (solid line) and 1 minus median specificity (dashed line) of CISs assigned by the observers in Figure 2. See Figures 1 and 2 for remainder of key

  • 1. Apley M. Bovine respiratory disease: pathogenesis, clinical signs, and treatment in lightweight calves. Vet Clin North Am Food Anim Pract 2006; 22: 399411.

    • Search Google Scholar
    • Export Citation
  • 2. Nicholas RA. Mycoplasma bovis: disease, diagnosis, and control. Res Vet Sci 2003; 4: 105112.

  • 3. Wittum TE, Woollen NE, Perino LJ, et al. Relationships among treatment for respiratory tract disease, pulmonary lesions evident at slaughter, and rate of weight gain in feedlot cattle. J Am Vet Med Assoc 1996; 209: 814818.

    • Search Google Scholar
    • Export Citation
  • 4. Gardner BA, Dolezal HG, Bryant LK, et al. Health of finishing steers: effects on performance, carcass traits, and meat tenderness. J Anim Sci 1999; 77: 31683175.

    • Search Google Scholar
    • Export Citation
  • 5. White BJ, Renter DG. Bayesian estimation of the performance of using clinical observations and harvest lung lesions for diagnosing bovine respiratory disease in post-weaned beef calves. J Vet Lab Diagn 2009; 21: 446453.

    • Search Google Scholar
    • Export Citation
  • 6. Hayes G, Mathews K, Kruth S, et al. Illness severity scores in veterinary medicine: what can we learn? J Vet Intern Med 2010; 24: 457466.

    • Search Google Scholar
    • Export Citation
  • 7. Dohoo I, Martin W, Stryhm H. Veterinary epidemiologic research. 2nd ed. Charlottetown, PE, Canada: VER Inc, 2009; 5: 9294.

  • 8. White BJ, Anderson DE, Renter DG, et al. Clinical, behavioral, and pulmonary changes following Mycoplasma bovis challenge in calves. Am J Vet Res 2012; 73: 490497.

    • Search Google Scholar
    • Export Citation
  • 9. Perino LJ, Apley M. Clinical trial design in feedlots. Vet Clin North Am Food Anim Pract 1998; 14: 243266.

  • 10. Fajt VR, Apley MD, Roth JA, et al. The effects of danofloxacin and tilmicosin on neutrophil function and lung consolidation in beef heifer calves with induced Pasteurella (Mannheimia) haemolytica pneumonia. J Vet Pharmacol Ther 2003; 26: 173179.

    • Search Google Scholar
    • Export Citation
  • 11. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159174.

  • 12. Kristensen E, Dueholm L, Vink D, et al. Within- and across-person uniformity of body condition scoring in Danish Holstein cattle. J Dairy Sci 2006; 89: 37213728.

    • Search Google Scholar
    • Export Citation
  • 13. Wagai J, Senga J, Fegan G, et al. Examining agreement between clinicians when assessing sick children. PloS One [serial online] 2009; 4:e4626. Available at www.plosone.org/article/info%3adoi%2F10journal.pone.0004626. Accessed Jan 21, 2012.

    • Search Google Scholar
    • Export Citation
  • 14. March S, Brinkmann J, Winkler C. Effect of training on the inter-observer reliability of lameness scoring in dairy cattle. Anim Welf 2007; 16: 131133.

    • Search Google Scholar
    • Export Citation
  • 15. Keegan KG, Dent EV, Wilson DA, et al. Repeatability of subjective evaluation of lameness in horses. Equine Vet J 2010; 42: 9297.

  • 16. Channon AJ, Walker AM, Pfau T, et al. Variability of Manson and Leaver locomotion scores assigned to dairy cows by different observers. Vet Rec 2009; 164: 388392.

    • Search Google Scholar
    • Export Citation
  • 17. Thompson PN, Stone A, Schultheiss WA. Use of treatment records and lung lesion scoring to estimate the effect of respiratory disease on growth during early and late finishing periods in South African feedlot cattle. J Anim Sci 2006; 84: 488498.

    • Search Google Scholar
    • Export Citation
  • 18. Schneider MJ, Tait RG Jr, Busby WD, et al. An evaluation of bovine respiratory disease complex in feedlot cattle: impact on performance and carcass traits using treatment records and lung lesion scores. J Anim Sci 2009; 87: 18211827.

    • Search Google Scholar
    • Export Citation
  • 19. Bryant LK, Perino LJ, Griffin D, et al. A method for recording pulmonary lesions of beef calves at slaughter, and the association of lesions with average daily gain. Bovine Pract 1999; 33: 163173.

    • Search Google Scholar
    • Export Citation

Advertisement

Precision and accuracy of clinical illness scores, compared with pulmonary consolidation scores, in Holstein calves with experimentally induced Mycoplasma bovis pneumonia

David E. Amrine DVM1, Brad J. White DVM, MS2, Robert Larson DVM, PhD3, David E. Anderson DVM, MS4, Derek A. Mosier DVM, PhD5, and Natalia Cernicchiaro DVM, PhD6
View More View Less
  • 1 Department of Diagnostic Medicine and Pathobiology, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506.
  • | 2 Department of Clinical Sciences, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506.
  • | 3 Department of Clinical Sciences, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506.
  • | 4 Department of Clinical Sciences, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506.
  • | 5 Department of Diagnostic Medicine and Pathobiology, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506.
  • | 6 Department of Diagnostic Medicine and Pathobiology, College of Veterinary Medicine, Kansas State University, Manhattan, KS 66506.

Abstract

Objective—To determine the precision of a clinical illness score (CIS) system for identification of clinical signs in calves with experimentally induced Mycoplasma bovis pneumonia and to evaluate the accuracy of CISs in relation to pulmonary consolidation scores assigned at necropsy.

Animals—178 Holstein bull calves that were 52 to 91 days of age at the time of pneumonia induction.

Procedures—5 trials involved calves challenged with M bovis and scheduled for euthanasia and necropsy 12 to 24 days afterward. Nine veterinarian observers with various degrees of experience simultaneously assigned CISs to calves within 48 hours before necropsy. The precision of the CIS system among observers was evaluated via the Cohen κ statistic. The accuracy of each observer's CISs relative to 6 cutoffs (≥ 5%, ≥ 10%, ≥ 15%, ≥ 20%, ≥ 25%, and ≥ 30%) of percentage pulmonary consolidation was determined by comparing prenecropsy CISs with the gross pulmonary consolidation scores assigned at necropsy. Estimates for sensitivity and specificity were calculated relative to the 6 pulmonary consolidation cutoffs.

Results—A slight level of agreement was evident among observers (κ range, 0.10 to 0.21 for the individual trials) and overall (κ = 0.16; 95% confidence interval, 0.10 to 0.24). Median sensitivity and specificity changed with pulmonary consolidation score cutoff. Median sensitivity for all observers ranged from 81.7% to 98.9%, and median specificity ranged from 80.8% to 94.9% over all cutoff values.

Conclusions and Clinical Relevance—Agreement among observers assigning CISs to calves was low; the accuracy of the CIS system in relation to that of pulmonary consolidation scoring varied with the severity of consolidation considered to represent bovine respiratory disease.

Abstract

Objective—To determine the precision of a clinical illness score (CIS) system for identification of clinical signs in calves with experimentally induced Mycoplasma bovis pneumonia and to evaluate the accuracy of CISs in relation to pulmonary consolidation scores assigned at necropsy.

Animals—178 Holstein bull calves that were 52 to 91 days of age at the time of pneumonia induction.

Procedures—5 trials involved calves challenged with M bovis and scheduled for euthanasia and necropsy 12 to 24 days afterward. Nine veterinarian observers with various degrees of experience simultaneously assigned CISs to calves within 48 hours before necropsy. The precision of the CIS system among observers was evaluated via the Cohen κ statistic. The accuracy of each observer's CISs relative to 6 cutoffs (≥ 5%, ≥ 10%, ≥ 15%, ≥ 20%, ≥ 25%, and ≥ 30%) of percentage pulmonary consolidation was determined by comparing prenecropsy CISs with the gross pulmonary consolidation scores assigned at necropsy. Estimates for sensitivity and specificity were calculated relative to the 6 pulmonary consolidation cutoffs.

Results—A slight level of agreement was evident among observers (κ range, 0.10 to 0.21 for the individual trials) and overall (κ = 0.16; 95% confidence interval, 0.10 to 0.24). Median sensitivity and specificity changed with pulmonary consolidation score cutoff. Median sensitivity for all observers ranged from 81.7% to 98.9%, and median specificity ranged from 80.8% to 94.9% over all cutoff values.

Conclusions and Clinical Relevance—Agreement among observers assigning CISs to calves was low; the accuracy of the CIS system in relation to that of pulmonary consolidation scoring varied with the severity of consolidation considered to represent bovine respiratory disease.

Bovine respiratory disease is commonly diagnosed in cattle on the basis of clinical signs, with the decision to treat typically made on the basis of a high rectal temperature.1 Mycoplasma bovis is an important pathogen in BRD and one of the causes of calf pneumonia.2 Although a 2-staged approach is often used to establish whether a calf has BRD, studies3,4 have revealed that > 50% of pulmonary lesions detected at slaughter are in calves that were never treated for respiratory disease. The sensitivity and specificity of clinical signs of illness followed by confirmatory rectal temperature for diagnosing BRD are reportedly only 62% and 63%, respectively.5 These findings suggest that considerable opportunities exist to improve diagnostic accuracy for respiratory disease in cattle.

A CIS system is commonly used in the clinical assessment of animals and quantification or assignment of values that correspond to the probability of a specific outcome.6 By having multiple observers evaluate and assign a CIS to the same calf, the agreement among observers can be used to determine the repeatability of that CIS system. Determining the repeatability (a measure of precision) of a diagnostic test allows for a better understanding of the variability among observers, which can be used when evaluating methods to improve agreement and, thereby, allows consistent evaluation of disease status. The accuracy of a test refers to how well the test identifies the true status of an individual and is based on sensitivity and specificity.7 In BRD, the true disease status can be assessed on the basis of the amount of pulmonary damage at a given time point.

Given that CISs are used to make decisions regarding treatment effectiveness, an estimate of the system's performance as a diagnostic test is necessary to interpret results from any BRD study in which CISs are used to define inclusion criteria, to allocate experimental units, or to classify outcome (eg, case vs noncase). The objectives of the study reported here were to determine the precision of a CIS system as a diagnostic tool to identify clinical signs of illness in calves with experimentally induced M bovis pneumonia and to evaluate the accuracy of that system in relation to pulmonary consolidation scoring at necropsy.

Materials and Methods

Animals—Five trials were conducted at the Kansas State University College of Veterinary Medicine between August 2010 and September 2011. Each trial was independent, but clinical observations were conducted in conjunction with procedures within each trial to elucidate the precision and accuracy of CIS assigned by multiple investigators relative to pulmonary consolidation scores. In total, 178 Holstein bull calves were enrolled in 5 trials (Appendix).

Calves were procured from a commercial calf grower, where they were individually housed in commercial hutches and fed a commercial milk replacer and calf starter ration until weaning. All calves were weaned and comingled on the farm of origin approximately 2 weeks prior to relocation to the Kansas State University Large Animal Research Center, with the exception of the calves in trial 3, which were weaned approximately 3 days prior to arrival. During the comingling phase, calves were fed only the starter ration and given free access to water. Calves in trials 4 and 5 received a series of M bovis vaccinesa prior to arrival at the research center.

Upon arrival, all calves received ceftiofur crystalline free acidb (6.6 mg/kg, SC, in the base of the ear). Calves in trials 1 and 2 were housed in open-air, dirt-floor pens with an attached, open-face shed on the north end (total area of each pen, 544 m2). In trial 3, calves were housed in an open-air, concrete-floor pen with an open-face shed attached to the north end. Calves in the remaining 2 trials (trials 4 and 5) were housed in open-air, dirt-floor pens (total area of each pen, approx 297 m2). A starter rationc was fed to calves in trials 1 and 2, and an alternative rationd was fed to calves in the remaining 3 trials. Free access to water and grass hay was made available to all calves in all trials. The study protocol was approved by the Kansas State University Institutional Animal Care and Use Committee.

M bovis challenge trials—Details of the sample collection procedures, laboratory protocols, and trial procedures used are described elsewhere.8 In all trials, calves were randomly assigned to an inoculation method or inoculation dose with the aid of a commercial software program.e The M bovis organismf used for pneumonia induction was the same for every trial. After the M bovis challenge, the same trained veterinarian (DEA) observed each calf twice daily and assigned a CIS until the calves were euthanized on the scheduled study completion date (Appendix). By trial design, calves with a CIS of 4 at any point during the trial were immediately euthanized and a necropsy was performed on the same day.

CIS system—The CIS system used was a modified version of one previously reported.9 Scores ranged from 1 to 4 as follows: 1 = usual behavior; 2 = slight illness (mild signs of depression or cough); 3 = moderate illness (severe signs of depression, labored breathing, or cough); and 4 = severe illness (moribund or failure to respond to human approach). Descriptions of each score were provided at the bottom of the scoring sheet so observers could reference these criteria when observing calves.

Multiple observer clinical illness scoring—Nine veterinarians with various degrees of experience were involved as observers in the assignment of CISs to calves over the course of the 5 trials. Four observers each had ≥ 14 years of large animal (bovine) clinical and research experience and held a faculty position at the Kansas State University College of Veterinary Medicine. The remaining observers were involved in postgraduate training in clinical or production large animal medicine.

To determine the precision of the CIS system, multiple observers evaluated each calf on the same day, 24 to 48 hours prior to the necropsy date (Appendix). In each trial, observers were provided with identical scoring sheets and asked to observe each calf and assign a CIS for each calf at each assessment point. All observers entered and exited the pen at approximately the same time (4:30 pm) and were unaware of the other observers' scores. Observers were free to move about the pen to visually inspect all calves.

Necropsy and pulmonary consolidation scoring—Calves in all trials were euthanized at a predetermined point after challenge (Appendix), in accordance with AVMA euthanasia guidelines regarding use of penetrating captive bolt.g Necropsy was performed on all calves, including gross examination of all major organ systems. The lungs and trachea were removed from each calf for pulmonary scoring, which was based on the percentage of each lung lobe with pneumonic change.10 Percentage pulmonary consolidation was calculated by the proportion of the lung that a given lung lobe represented, and the percentage consolidation for each lung lobe was multiplied by the fraction each lobe represented of the whole lung. Lobe values were totaled and multiplied by 100 to yield the reported consolidation score. The accuracy of the CIS system relative to pulmonary consolidation scoring was determined by comparing CISs assigned within 48 hours prior to necropsy with the gross pulmonary consolidation scores assigned at necropsy.

Statistical analysis—The precision (or agreement) of CISs among observers was calculated on an individual trial basis and overall (all trial) as the Cohen κ statistic and associated 95% CIs with the aid of statistical software.h Observers' scores for each calf were transformed into a dichotomous variable, with 0 representing an apparently healthy state (CIS, 1) and 1 representing a diseased state (CIS > 1) for all κ calculations. The following scale11 was used to interpret values of κ: ≤ 0, poor agreement; 0.01 to 0.20, slight agreement; 0.21 to 0.40, fair agreement; 0.41 to 0.60, moderate agreement; 0.61 to 0.80, substantial agreement; and 0.81 to 1.00, almost perfect agreement.

To evaluate the accuracy of the CIS system versus the pulmonary consolidation system, CISs assigned within 48 hours prior to necropsy were compared with pulmonary consolidation scores obtained at necropsy, with consolidation scores considered to represent a calf's true disease status (ie, the reference standard). Because no standard exists for defining a percentage of pulmonary involvement that would require medical intervention, 6 cutoff values were used, ranging in intervals of 5% from ≥ 5% to ≥ 30%.

Pulmonary consolidation scores for each calf were considered positive (greater than or equal to the cutoff) or negative (less than the cutoff) at each of the 6 cutoff values. A dichotomous variable was created and populated with a value of 1 when a calf's pulmonary consolidation score was greater than or equal to the cutoff, otherwise the variable contained a value of zero. Clinical illness score was treated as a dichotomous variable (healthy vs diseased) as well, as described for precision calculations. For each calf, comparisons were then made of the CIS dichotomous value with each of the pulmonary consolidation cutoffs to determine the accuracy of the CIS system at each cutoff. A calf with a CIS status of healthy but a pulmonary consolidation status of diseased was considered to have a false-negative result, and one with a CIS status of diseased but a pulmonary consolidation status of diseased was considered to have a false-positive result.

Statistical softwarei was used to perform generalized linear mixed modeling (binomial distribution and logit link function) to estimate the probability of misclassification of disease state. The probability of an observer assigning a calf a false-positive or false-negative result was considered the outcome of interest at each cutoff value. Repeated measures on calf and trial were included in each model to account for a lack of independence between samples. The sensitivity of the CIS system was subsequently calculated by subtracting from 1 the probability of an observer falsely scoring a calf as healthy at a given cutoff. Specificity was calculated by subtracting from 1 the probability of an observer falsely scoring a calf as diseased at a given pulmonary consolidation cutoff. Sensitivity and specificity were estimated with data from only trials 2 through 5 because scoring of these calves was done in the afternoon before necropsy, contrary to trial 1, in which scoring was done several days prior to necropsy. Because data calculated for sensitivity and specificity at each cutoff were not normally distributed, median (range) values are reported.

Results

Animals—Clinical illness scores were assigned to 178 calves over 5 trials by 9 observers. The same 4 observers scored calves in all 5 trials, and the remaining 5 observers scored calves in 1 to 3 trials on the basis of their availability. No calf in any trial received a CIS > 3. In trials 2 through 5, 87 of 154 (56%) calves had pulmonary consolidation scored as ≥ 5% and 24 of 154 (16%) received a score ≥ 30% (Figure 1).

Figure 1—
Figure 1—

Percentage of Holstein calves inoculated with Mycoplasma bovis that had pulmonary consolidation scores at various cutpoints as determined at necropsy (n = 154). Pulmonary consolidation for each lung lobe was assigned a percentage by a veterinary pathologist. Total pulmonary consolidation scores were calculated on the basis of a reported formula.10

Citation: American Journal of Veterinary Research 74, 2; 10.2460/ajvr.74.2.310

CIS precision—Precision, or agreement beyond chance, for all 9 observers over all trials was 0.16 (95% CI, 0.10 to 0.24), indicating slight to fair agreement (Table 1). Accuracy of the CIS system relative to the pulmonary consolidation scoring system was assessed via the calculated sensitivity and specificity at each pulmonary consolidation cutoff. The median (range) calculated sensitivity for all observers ranged from 81.7% (55.4% to 96.4%) at the ≥ 5% consolidation cutoff to 98.9% (93.9% to 99.8%) at the ≥ 30% consolidation cutoff (Figure 2). Median (range) specificity also varied by pulmonary consolidation score cutoff and ranged from 94.9% (81.3% to 97.3%) at the ≥ 5% cutoff to 80.8% (48.5% to 93.8%) at the ≥ 30% cutoff (Figures 3 and 4).

Table 1—

Median (range) pulmonary consolidation scores and agreement (95% CI) with CISs in calves with experimentally induced Mycoplasma bovis pneumonia in 5 trials that differed in dates of score assignment.

TrialPulmonary consolidation score (%)Agreement (κ)
1 (n = 24)2.4 (0–19.5)0.16 (0.03–0.43)
2 (n = 42)6.3 (0–13.4)0.13 (0.05–0.28)
3 (n = 16)1.8 (0.2–45.2)0.21 (0.05–0.47)
4 (n = 43)6.3 (0.2–47.4)0.17 (0.05–0.37)
5 (n = 53)6.1 (0.2–50.3)0.10 (0.01–0.25)
Overall agreement for all trials0.16 (0.10–0.24)

— = Not applicable.

See Appendix for information on how the 5 trials differed.

Figure 2—
Figure 2—

Box-and-whisker plots of estimated sensitivity values for CISs assessed in calves (n = 154) by 8 observers at each pulmonary consolidation score cutoff. Boxes represent the 25th and 75th quartiles, and the horizontal line within the boxes represents the median. Whiskers represent the minimum and maximum values. See Figure 1 for remainder of key.

Citation: American Journal of Veterinary Research 74, 2; 10.2460/ajvr.74.2.310

Figure 3—
Figure 3—

Box-and-whisker plots of estimated specificity values for CISs assessed by the observers represented in Figure 2. See Figures 1 and 2 for remainder of key.

Citation: American Journal of Veterinary Research 74, 2; 10.2460/ajvr.74.2.310

Figure 4—
Figure 4—

Graph of median sensitivity (solid line) and 1 minus median specificity (dashed line) of CISs assigned by the observers in Figure 2. See Figures 1 and 2 for remainder of key

Citation: American Journal of Veterinary Research 74, 2; 10.2460/ajvr.74.2.310

Discussion

Nine veterinarians with various backgrounds and experience participated in assessing the precision and accuracy of CISs relative to pulmonary consolidation scores assigned at necropsy in calves inoculated with M bovis. Overall, the precision or agreement among observers was poor; accuracy varied by the degree of pulmonary consolidation chosen to identify a truly diseased calf.

In the absence of a gold standard, agreement between observers is used to estimate the precision of a test.12,13 The interobserver agreement among all observers for all trials in our study was considered slight as defined elsewhere,11 indicating the repeatability of the CIS evaluated as a diagnostic test was limited among observers. All observers were veterinarians; therefore, they all had prior training that should have biased them toward correctly identifying a calf as clinically ill or not. Our results did not support the hypothesis that these observers scored the calves similarly. No effort was made to train observers on the scoring system, and it is possible that observers with more experience than the others at identifying ill cattle scored the calves on the basis of previous experiences more than on the CIS system.

Multiple observers are often called upon to diagnose illness on farms, and our results suggested that the CIS system lacks precision as a diagnostic test. Improvement of the case definition of a truly ill calf along with training of observers might improve agreement, as was found in a study14 in which multiple observers evaluated lameness in cattle. Because there were multiple raters per calf and the number of raters was not consistent across trials, no attempt was made to determine whether an individual observer's bias affected κ values through use of bias-adjusted κ values.7 The κ statistic has been used in veterinary medicine to estimate the agreement between clinicians in relation to lameness15,16; however, to our knowledge, interobserver agreement with regard to clinical illness scoring for the diagnosis of induced respiratory disease in calves has not been reported.

Assessment of clinical signs of illness to detect respiratory disease in cattle is generally practiced, but few reports exist to quantify the accuracy of this method versus pulmonary consolidation scoring. Two studies3,17 revealed that nearly 70% of calves with lesions at slaughter had never received treatment for respiratory disease, suggesting that not all calves with respiratory disease can be identified by clinical signs alone. However, in those studies, the temporal relationship between clinical signs of illness and pulmonary lesions was not known. In another study,5 the diagnostic sensitivity and specificity of use of signs of clinical illness followed by rectal temperature to diagnose disease in sick cattle were 61.8% and 62.8%, respectively. In our study, at the ≥ 5% cutoff, median sensitivity (81.7%) and specificity (94.9%) were higher than previous estimates, although the observers were all veterinarians and the study population was significantly different from that in the other study5 in regard to age and breed.

Median sensitivity (81.7%) was lowest at the ≥ 5% cutoff, where the range of values (55.5% to 96.4%) among observers was the largest. A threshold of pulmonary damage at which BRD is subclinical likely exists. Given the large increase in median sensitivity between the 5% (81.7%) and ≥ 10% (95.1%) cutpoints, several calves made the transition from subclinical to clinical disease and observers were more likely to identify the calves as clinically ill. As the cutoff for pulmonary consolidation became ≥ 10%, calves were generally more clinically ill. As calves' clinical signs of illness worsened, all observers more accurately identified these calves as diseased, which led to higher sensitivity values with narrower ranges of values among observers. The objective of the study was not to determine a specific cutpoint beyond which treatment is no longer beneficial because none of the calves were treated after pneumonia induction. Instead, an objective was to highlight some challenges inherent to a subjective method of identifying and quantifying disease in calves.

Median specificity (94.9%; range, 81.3% to 97.3%) was highest at the ≥ 5% cutoff, which indicated that observers less commonly falsely identified calves as having disease, compared with at other cutoff values. One explanation for this is that the lowest cutoff resulted in classification of > 50% of all calves as sick; therefore, fewer healthy calves were available to be falsely identified as disease positive in relation to higher pulmonary consolidation cutoff values. All calves were challenged with M bovis, and previous work8 with this challenge method yielded highly variable degrees of pulmonary consolidation and clinical illness. Therefore, although observers were aware that calves had been challenged, they had no reason to believe all calves would be ill immediately prior to necropsy.

A tradeoff in sensitivity and specificity was identified as the cutoff for determining a truly diseased calf increased. When deciding on an optimal cutoff for a diagnostic test, consideration should be given to the importance of false-negative and false-positive test results.7 When the extent of BRD in a herd is measured through clinical observation (as is commonly done in clinical trials), a test with imperfect sensitivity would not identify all truly ill animals. No previous studies have been conducted to specifically evaluate the impact of false negative test results, but several investigators3,18,19 found a negative association between the presence of pulmonary lesions at slaughter and average daily gain. Conversely, if calves are falsely identified as disease positive by a test with < 100% specificity, then the extent of disease will be overestimated, the effectiveness of preventative treatment will be falsely underestimated, and the effectiveness of disease treatment will be falsely overestimated.

Limitations to the present study existed that weaken its external validity. All calves used were young Holstein bulls inoculated with M bovis. Although observers knew that all calves had been exposed to M bovis and their scores may have been influenced by this knowledge, not all calves responded similarly and observers did not believe all calves would become sick because of previous work with this challenge model. Observers were aware that some calves in trials 4 and 5 had received prechallenge treatment (vaccination) but were not aware of treatment status for individual calves.

ABBREVIATIONS

BRD

Bovine respiratory disease

CI

Confidence interval

CIS

Clinical illness score

a.

CEVA/Biomune, Lenexa, Kan.

b.

Excede, Pfizer Animal Health, New York, NY.

c.

Herd Maker Supreme B90, Land O' Lakes, Shoreview, Minn.

d.

Calf Grower B-68 Medicated, Manhattan KS Coop, Manhattan, Kan.

e.

Excel 2010, Microsoft Corp, Redmond, Wash.

f.

Provided by CEVA/Biomune, Lenexa, Kan.

g.

Koch Magnum 0.25 Stunner, KOCH Supplies Inc, Kansas City, Mo.

h.

Stata/MP, version 12, StataCorp LP, College Station, Tex.

i.

PROC GLIMMIX, SAS, version 9.2, SAS Institute Inc, Cary, NC.

References

  • 1. Apley M. Bovine respiratory disease: pathogenesis, clinical signs, and treatment in lightweight calves. Vet Clin North Am Food Anim Pract 2006; 22: 399411.

    • Search Google Scholar
    • Export Citation
  • 2. Nicholas RA. Mycoplasma bovis: disease, diagnosis, and control. Res Vet Sci 2003; 4: 105112.

  • 3. Wittum TE, Woollen NE, Perino LJ, et al. Relationships among treatment for respiratory tract disease, pulmonary lesions evident at slaughter, and rate of weight gain in feedlot cattle. J Am Vet Med Assoc 1996; 209: 814818.

    • Search Google Scholar
    • Export Citation
  • 4. Gardner BA, Dolezal HG, Bryant LK, et al. Health of finishing steers: effects on performance, carcass traits, and meat tenderness. J Anim Sci 1999; 77: 31683175.

    • Search Google Scholar
    • Export Citation
  • 5. White BJ, Renter DG. Bayesian estimation of the performance of using clinical observations and harvest lung lesions for diagnosing bovine respiratory disease in post-weaned beef calves. J Vet Lab Diagn 2009; 21: 446453.

    • Search Google Scholar
    • Export Citation
  • 6. Hayes G, Mathews K, Kruth S, et al. Illness severity scores in veterinary medicine: what can we learn? J Vet Intern Med 2010; 24: 457466.

    • Search Google Scholar
    • Export Citation
  • 7. Dohoo I, Martin W, Stryhm H. Veterinary epidemiologic research. 2nd ed. Charlottetown, PE, Canada: VER Inc, 2009; 5: 9294.

  • 8. White BJ, Anderson DE, Renter DG, et al. Clinical, behavioral, and pulmonary changes following Mycoplasma bovis challenge in calves. Am J Vet Res 2012; 73: 490497.

    • Search Google Scholar
    • Export Citation
  • 9. Perino LJ, Apley M. Clinical trial design in feedlots. Vet Clin North Am Food Anim Pract 1998; 14: 243266.

  • 10. Fajt VR, Apley MD, Roth JA, et al. The effects of danofloxacin and tilmicosin on neutrophil function and lung consolidation in beef heifer calves with induced Pasteurella (Mannheimia) haemolytica pneumonia. J Vet Pharmacol Ther 2003; 26: 173179.

    • Search Google Scholar
    • Export Citation
  • 11. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159174.

  • 12. Kristensen E, Dueholm L, Vink D, et al. Within- and across-person uniformity of body condition scoring in Danish Holstein cattle. J Dairy Sci 2006; 89: 37213728.

    • Search Google Scholar
    • Export Citation
  • 13. Wagai J, Senga J, Fegan G, et al. Examining agreement between clinicians when assessing sick children. PloS One [serial online] 2009; 4:e4626. Available at www.plosone.org/article/info%3adoi%2F10journal.pone.0004626. Accessed Jan 21, 2012.

    • Search Google Scholar
    • Export Citation
  • 14. March S, Brinkmann J, Winkler C. Effect of training on the inter-observer reliability of lameness scoring in dairy cattle. Anim Welf 2007; 16: 131133.

    • Search Google Scholar
    • Export Citation
  • 15. Keegan KG, Dent EV, Wilson DA, et al. Repeatability of subjective evaluation of lameness in horses. Equine Vet J 2010; 42: 9297.

  • 16. Channon AJ, Walker AM, Pfau T, et al. Variability of Manson and Leaver locomotion scores assigned to dairy cows by different observers. Vet Rec 2009; 164: 388392.

    • Search Google Scholar
    • Export Citation
  • 17. Thompson PN, Stone A, Schultheiss WA. Use of treatment records and lung lesion scoring to estimate the effect of respiratory disease on growth during early and late finishing periods in South African feedlot cattle. J Anim Sci 2006; 84: 488498.

    • Search Google Scholar
    • Export Citation
  • 18. Schneider MJ, Tait RG Jr, Busby WD, et al. An evaluation of bovine respiratory disease complex in feedlot cattle: impact on performance and carcass traits using treatment records and lung lesion scores. J Anim Sci 2009; 87: 18211827.

    • Search Google Scholar
    • Export Citation
  • 19. Bryant LK, Perino LJ, Griffin D, et al. A method for recording pulmonary lesions of beef calves at slaughter, and the association of lesions with average daily gain. Bovine Pract 1999; 33: 163173.

    • Search Google Scholar
    • Export Citation

Appendix

Trial characteristics in a 5-trial study to determine the usefulness of a CIS system for BRD diagnosis in Holstein bull calves inoculated with Mycoplasma bovis.

VariableTrial 1Trial 2Trial 3Trial 4Trial 5
No. of calves2442164353
No. of observers78755
Observation day*612231111
Necropsy day*1413 and 14241213
Time of yearSept–DecJan–MarchJune–AugJune–AugJune–Sept

Days are relative to the day on which M bovis pneumonia was experimentally induced (day 0).

Contributor Notes

This manuscript represents a portion of a dissertation submitted by the senior author to the Kansas State University Department of Diagnostic Medicine and Pathobiology as partial fulfillment of the requirements for a Doctor of Philosophy degree.

Supported in part by CEVA/Biomune.

Presented in abstract form at the Phi Zeta Day of Kansas State University, Manhattan, Kan, March 2012.

The authors thank Drs. Matt Miesner, Brandon Fraser, Carrie Wheeler, and Amanda Hartnack for assistance in data collection.

Address correspondence to Dr. White (bwhite@vet.k-state.edu).