Analytic performance evaluation of a veterinary-specific ELISA for measurement of serum cortisol concentrations of dogs

Michael B. Lane Department of Small Animal Clinical Sciences, College of Veterinary Medicine, University of Tennessee, Knoxville, TN 37996.

Search for other papers by Michael B. Lane in
Current site
Google Scholar
PubMed
Close
 DVM
,
Bente Flatland Department of Biomedical and Diagnostic Sciences, College of Veterinary Medicine, University of Tennessee, Knoxville, TN 37996.

Search for other papers by Bente Flatland in
Current site
Google Scholar
PubMed
Close
 DVM, MS
,
Shelly J. Olin Department of Small Animal Clinical Sciences, College of Veterinary Medicine, University of Tennessee, Knoxville, TN 37996.

Search for other papers by Shelly J. Olin in
Current site
Google Scholar
PubMed
Close
 DVM
,
Kellie A. Fecteau Department of Biomedical and Diagnostic Sciences, College of Veterinary Medicine, University of Tennessee, Knoxville, TN 37996.

Search for other papers by Kellie A. Fecteau in
Current site
Google Scholar
PubMed
Close
 PhD
,
Markus Rick Department of Pathobiology and Diagnostic Investigation, College of Veterinary Medicine, Michigan State University, East Lansing, MI 48824.

Search for other papers by Markus Rick in
Current site
Google Scholar
PubMed
Close
 DVM, PhD
, and
Luca Giori Department of Biomedical and Diagnostic Sciences, College of Veterinary Medicine, University of Tennessee, Knoxville, TN 37996.

Search for other papers by Luca Giori in
Current site
Google Scholar
PubMed
Close
 DVM, PhD

Abstract

OBJECTIVE To investigate the precision of an ELISA for measurement of serum cortisol concentration (SCC) in dogs, assess agreement between this ELISA and 2 validated chemiluminescence assays (CLAs), and evaluate the clinical implications of any bias associated with this ELISA when measuring SCC in dogs.

DESIGN Evaluation study.

SAMPLE 75 stored, frozen serum samples from client-owned dogs.

PROCEDURES Enzyme-linked immunosorbent assay precision was evaluated by measuring SCC of pooled serum samples. Agreement with standard methods was evaluated with Spearman rank correlation, Passing-Bablok regression, and Bland-Altman analysis to compare SCCs obtained with the ELISA and the 2 CLAs. An error grid was used to evaluate identified bias.

RESULTS Within-laboratory coefficients of variation for pooled serum samples with low, medium, and high SCCs were 21.4%, 28.9%, and 13.0%, respectively. There was a high correlation between ELISA results (for all samples combined) and results of the 2 CLAs (CLA 1, r = 0.96; CLA 2, r = 0.97), but constant and proportional biases between the ELISA and CLAs were present at all concentrations. Clinically important disagreement between ELISA results and CLA results occurred in 16 of 63 (25%) samples, particularly with low and high SCCs.

CONCLUSIONS AND CLINICAL RELEVANCE Results suggested that the rate of clinical disagreement between the ELISA and CLAs was sufficiently high to recommend that equivocal results obtained with the ELISA be confirmed by a reference laboratory. Further evaluation of analytic performance of the ELISA should focus on samples with very high and very low SCCs.

Abstract

OBJECTIVE To investigate the precision of an ELISA for measurement of serum cortisol concentration (SCC) in dogs, assess agreement between this ELISA and 2 validated chemiluminescence assays (CLAs), and evaluate the clinical implications of any bias associated with this ELISA when measuring SCC in dogs.

DESIGN Evaluation study.

SAMPLE 75 stored, frozen serum samples from client-owned dogs.

PROCEDURES Enzyme-linked immunosorbent assay precision was evaluated by measuring SCC of pooled serum samples. Agreement with standard methods was evaluated with Spearman rank correlation, Passing-Bablok regression, and Bland-Altman analysis to compare SCCs obtained with the ELISA and the 2 CLAs. An error grid was used to evaluate identified bias.

RESULTS Within-laboratory coefficients of variation for pooled serum samples with low, medium, and high SCCs were 21.4%, 28.9%, and 13.0%, respectively. There was a high correlation between ELISA results (for all samples combined) and results of the 2 CLAs (CLA 1, r = 0.96; CLA 2, r = 0.97), but constant and proportional biases between the ELISA and CLAs were present at all concentrations. Clinically important disagreement between ELISA results and CLA results occurred in 16 of 63 (25%) samples, particularly with low and high SCCs.

CONCLUSIONS AND CLINICAL RELEVANCE Results suggested that the rate of clinical disagreement between the ELISA and CLAs was sufficiently high to recommend that equivocal results obtained with the ELISA be confirmed by a reference laboratory. Further evaluation of analytic performance of the ELISA should focus on samples with very high and very low SCCs.

A long with appropriate clinical signs, accurate measurement of SCC is necessary for definitive diagnosis of hypo- and hyperadrenocorticism in dogs when performing dynamic diagnostic tests, such as the low-dose dexamethasone suppression and ACTH stimulation tests.1,2 For SCC measurement in dogs, an RIA is the historical gold standard,3 but this assay has largely been replaced by CLAs in most reference laboratories because of inherent disadvantages of the RIA method (eg, radioactive tracer component, short reagent shelf-life, and disposal challenges) and because the manufacturer has discontinued production of the most commonly used RIA.4,a Furthermore, CLAs are validated, specific, and free from interference and provide results that strongly agree with RIA results.4,5

An enticing alternative to CLAs, which require sample submission to a reference laboratory, is an ELISA that can be performed, analyzed, and interpreted with a proprietary analyzer in veterinary practices within minutes of sample acquisition. A veterinary-specific ELISA for the measurement of SCC in dogs has been available to practitioners for over a decade; however, analytic performance data for this in-house assay have not been published in the peer-reviewed literature. Current expert recommendations suggest that use of the ELISA to measure SCC should be reserved for emergency cases and that results should be verified by a reference laboratory whenever they would lead to a change in treatment.6 Although test kits and instruments undergo analytic performance evaluation during development and manufacturing, it is important for testing sites (eg, clinics and laboratories) to perform independent verification of analytic performance to ensure that instruments and methods meet the manufacturer's performance claims and are suitable for their intended clinical purposes.7

The purposes of the study were to investigate the precision of a veterinary-specific ELISA for measurement of SCC in dogs, assess agreement between this ELISA and 2 validated CLAs (as an evaluation of ELISA accuracy), and evaluate the clinical implications of any bias associated with this ELISA when measuring SCC in dogs. We hypothesized that the ELISA would have good precision over a wide range of SCCs, results of the ELISA would have high correlation and good agreement with results of the CLAs, and any identified bias between the ELISA and CLAs would not impact clinical decisions.

Materials and Methods

Stored (−80°C) samples of excess serum remaining from diagnostic samples that had been submitted by practitioners over the previous 3 months to the Clinical Endocrinology Service at the University of Tennessee College of Veterinary Medicine for measurement of cortisol, mineralocorticoid, or sex steroid hormone concentrations in dogs were used in the study. Samples were semiquantitatively evaluated by 2 investigators (MBL and LG) for the presence of lipemia, hemolysis, and icterus on the basis of current guidelines.8 Results were recorded as absent, trace, or present, with present graded on a scale from 1 to 4.

A commercially available veterinary-specific ELISA test kitb in combination with a proprietary analyzer,c which was inspected on-site by the manufacturer's technical specialist prior to beginning the study, was used to measure SCCs of the samples. The ELISA test kits were designed for use with the analyzer, and the analyzer performed image analysis on the test kits by taking digital pictures as the test kit results developed and then applying algorithms according to the appropriate protocol. For measurement of SCC in dogs, there were 2 protocols from which the instrument operator could select on the basis of the type of testing being performed or clinical suspicions. The baseline-cortisol protocol had a dynamic SCC range from 0.5 to 10 μg/dL (13.8 to 275.9 nmol/L) and was used to measure SCC prior to dexamethasone or ACTH administration for their respective tests. The required sample size for this protocol was 100 μL of serum. The Cushing disease–suspected protocol had a dynamic SCC range from 2.5 to 30 μg/dL and was used to measure SCC when testing dogs suspected of having hyperadrenocorticism. The required sample size for this protocol was 25 μL of serum.

Reference methods for analytic evaluation of the ELISA were 2 CLAs (CLA 1d,e and CLA 2f,g) performed with similar automated platforms from the same manufacturer. The analyzers for the CLAs were used to perform a solid-phase, competitive chemiluminescent enzyme immunoassay on the samples. The procedure involved competitive binding of labeled antigens (in this case, cortisol labeled with alkaline phosphatase) versus nonlabeled antigens (cortisol in the test sample) to the limited number of antibody binding sites located on plastic beads (anti-cortisol antibody–coated solid phase). The amount of labeled antigen bound to the plastic beads was determined by adding a chemiluminescent substrate that generated light after reacting with alkaline phosphatase. The amount of light emitted was inversely proportional to the amount of nonlabeled antigen that was bound, allowing cortisol concentration in the test sample to be calculated algorithmically. Precision of each CLA method was calculated on the basis of control data for the previous 40 days with the following formula:

CV = 100 × (SD/mean cortisol concentration).

Precision of the ELISA

Precision of the ELISA was investigated to determine random analytic error associated with the ELISA measuring SCC in dogs. Stored serum samples were selected on the basis of SCC measured when the samples were originally submitted. Samples with an SCC between 0.5 and 6.0 μg/dL (13.8 and 165.5 nmol/L), between 7.0 and 17.0 μg/dL (193.1 and 469.0), and between 18.0 and 30.0 μg/dL (496.6 and 827.7 nmol/L) were mixed to create pooled samples with low, medium, and high SCCs, respectively. The SCC of each pool was confirmed with CLA 1 to ensure that it fell within the desired range. Each pool was then divided into 15 aliquots, which were frozen at −80°C until the time of testing with the ELISA to avoid variations in the number of freeze-thaw cycles. The SCC of thawed aliquots was measured in triplicate with the ELISA on 5 consecutive days, in accordance with established guidelines for precision evaluation.9 On the first day of sample analysis, the SCC of each pool was reverified with CLA 1 to ensure that the concentration was within the desired range.

All ELISA test kits were from the same lot number, and all tests were performed in accordance with the manufacturer's instructions10 by a single investigator (MBL) who was blinded to the concentrations of the pooled samples. Cortisol concentrations in samples from the low SCC pool were measured with the baseline-cortisol protocol, whereas concentrations in samples from the medium and high SCC pools were measured with the Cushing disease–suspected protocol. When errors in the formation and development phase of the antigen-antibody–mediated color reaction product of the assay occurred, causing an error result, the test was repeated with a new ELISA kit until an SCC was reported by the analyzer.

Method comparison

Results of the ELISA were compared with results of the reference laboratory methods (CLA 1 and CLA 2) to investigate bias (systematic analytic error) of the ELISA. Stored serum samples were selected on the basis of SCC measured when the samples were originally submitted and specimen volume (to ensure sufficient volume for measurement of SCC by all 3 study methods). Twenty samples with a low SCC (0.5 to 6.0 μg/dL [13.8 to 165.5 nmol/L]), 20 samples with a medium SCC (7.0 to 17.0 μg/dL [193.1 to 469.0 nmol/L]), and 20 samples with a high SCC (> 18.0 μg/dL [> 496.6 nmol/L]) were chosen to represent clinically relevant SCCs on the basis of decision thresholds established by the ELISA manufacturer.

To avoid variation in the number of freeze-thaw cycles, all samples were thawed once and divided into aliquots, which were then frozen at −80°C until the time of testing. Aliquot volumes were slightly greater than the cumulative sample volume requirements for the 3 study methods.

For the ELISA, all samples were measured in accordance with the manufacturer's instructions10 and were randomized for measurement in duplicate by a single, blinded investigator (MBL). To avoid any bias that might have occurred as a result of clinical suspicion of a specific adrenocortical disorder (hypo- vs hyperadrenocorticism), samples with low and medium cortisol concentrations were randomized for analysis with the 2 analytic protocols (baseline-cortisol protocol, SCC of 0.5 to 10.0 μg/dL; Cushing disease–suspected protocol, SCC of 2.5 to 30.0 μg/dL) on the ELISA analyzer.8 Samples in the high SCC group were randomized for measurement in duplicate with only the Cushing disease–suspected protocol because the SCC of these samples would have exceeded the ELISA's reportable range for the baseline protocol.

For the CLAs, frozen aliquots were randomized and measured in duplicate at the University of Tennessee Veterinary Medical Center Clinical Endocrinology Service by a single laboratory technician who was also blinded to the original SCCs. Measurements with CLA 1 (previously validated for measurement of SCC in dogs4) were performed on the same day and in the same room as the ELISA measurements. Frozen aliquots of the samples were then mailed overnight on dry ice to the Michigan State University Diagnostic Center for Population and Animal Health, where samples were stored at −80°C until the time of test measurement at that facility. All of the measurements at this second reference laboratory were made with CLA 2g (also previously validated for measurement of SCC in dogs5), occurred within 1 week of receipt, and were performed in duplicate by a single laboratory technician, also blinded to the original SCCs.

Statistical analysis

For the precision study, the CV of the ELISA was calculated with the grand mean of the SCCs (ie, mean of the SCCs obtained for each pool over all days) and sr or sl by use of the formula CV = (100 × s)/ (grand mean SCC). Taken from the relevant Clinical and Laboratory Standards Institute guideline, sr and sl replace older terminology for random analytic error (eg, within-day and within-run and between-day and between-run error).9

For the method comparison study, a Shapiro-Wilk test was used to determine whether data were normally distributed. The correlation between methods was calculated with the Spearman rank correlation method because not all data were normally distributed. Correlation results were categorized as very high (r > 0.90), high (r = 0.70 to 0.90), moderate (r = 0.50 to 0.70), or low (r = 0.30 to 0.50).11 Agreement between methods was evaluated with Passing-Bablok regression analysis. When the y-intercept of the regression line was 0, or the 95% CI for the y-intercept of the regression line contained 0, it was accepted that no constant systematic bias was present. When the slope of the regression line was 1, or the 95% CI for the slope of the regression line contained 1, it was accepted that no proportional systematic bias was present. Agreement was further evaluated by use of Bland-Altman plots, with bias defined as the mean difference between methods. Statistical analyses were performed with commercially available software.h Values of P < 0.05 were considered significant.

Error grid analysis

An error grid was created with decision thresholds established by the ELISA manufacturer for interpretation of SCC following ACTH stimulation testing.10 For the purposes of error grid construction, the assumption was made that data would represent SCC as measured after an ACTH stimulation test in a dog with clinical signs suggestive of hypo- or hyperadrenocorticism. Briefly, SCCs < 2.0 μg/dL (55.2 nmol/L) and between 2.0 and 6.0 μg/dL (55.2 to 165.5 nmol/L) were considered consistent with and equivocal for hypoadrenocorticism, respectively; SCCs between 6.0 and 18.0 μg/dL (165.5 to 496.6 nmol/L) were considered normal; and SCCs between 18.0 and 22.0 μg/dL (496.6 and 607.0 nmol/L) and > 22.0 μg/dL (> 607.0 nmol/L) were considered equivocal and consistent with hyperadrenocorticism, respectively. Samples with SCC > 30.0 μg/dL were not depicted; however, those values fell within the region of clinical agreement and were used to calculate overall clinical agreement between the methods. Three zones (A, B, and C) were then overlaid on the grid. Results in zone A, the zone of clinical agreement, would lead to the same clinical decision regardless of any bias between methods. Results in zone B, the zone of clinical disagreement, would lead to delay in a diagnosis or inappropriately conclusive or inconclusive results because of bias. Results in zone C, the zone of clinical misdiagnosis, would lead to a misdiagnosis or inappropriate clinical decision because of bias. Once the error grid was constructed, data from the method comparison study were plotted on the grid. Mean SCC of the duplicate ELISA results (y-axis) was plotted against the mean SCC of both CLA methods (x-axis). In other words, each x-axis coordinate was the average of all 4 CLA results for that specimen (2 replicates from each reference laboratory). For the purposes of this comparison, only data generated by the ELISA performed with the more appropriate protocol were used. For instance, if the SCC was > 10.0 μg/dL, only the results obtained with the Cushing disease–suspected protocol were included. The number of data points in each zone was then counted, and the number of samples in each zone was calculated as a percentage of all samples.

Results

For the entire study, stored excess serum from 75 diagnostic samples from dogs was used, including 12 samples used to evaluate the precision of the ELISA (6, 2, and 4 samples for pooled samples with low, medium, and high SCC, respectively) and 63 samples used to compare ELISA results with results of the 2 CLAs. Coefficients of variation for CLA 1 were 8.3% at low SCC (mean SCC, 6.3 μg/dL [173.8 nmol/L]) and 6.7% at high SCC (mean SCC, 22.2 μg/dL [612.5 nmol/L]). Coefficients of variation for CLA 2 were 6.6% at low SCC (mean SCC, 3.9 μg/dL [106.5 nmol/L]), 5.8% at medium SCC (mean SCC, 21.8 μg/dL [600.9 nmol/L]), and 4.3% at high SCC (mean SCC, 33.8 μg/dL [933.5 nmol/L]).

Precision of the ELISA

Samples included in the pooled serum samples had, at most, trace lipemia or hemolysis, alone or in combination. One result for the pooled sample with a high SCC was > 30.0 μg/dL, which indicated that the concentration was above the analyzer's reportable range. For the purpose of precision calculations, this result was assigned an SCC of 30.0 μg/dL. Grand mean SCC for the pooled samples with low, medium, and high SCCs were 3.0 μg/dL (88.8 nmol/L), 10.6 μg/dL (292.5 nmol/L), and 24.7 μg/dL (681.5 nmol/L), respectively.

Values of sr for pooled samples with low, medium, and high SCCs were 0.47, 1.91, and 1.69 μg/dL, respectively, and the corresponding CVs were 15.5%, 17.9%, and 6.81%. Values of sl for pooled samples with low, medium, and high SCCs were 0.64, 3.08, and 3.22 μg/dL, respectively, and the corresponding CVs were 21.4%, 28.9%, and 13.0%.

Method comparison study

Of the 60 stored serum samples initially selected for the method comparison study, 9 (4 with low SCC, 1 with medium SCC, and 4 with high SCC) were excluded because of a delay in processing that led to variation in the number of freeze-thaw cycles that could have added a source of variability. Additionally, 3 samples with high SCCs were excluded because repeated errors during the ELISA development phase prompted repeated measurements until insufficient specimen volume remained. To compensate for these losses, an additional 15 samples with high SCC were included in the study and handled in the same way as the original samples. Therefore, 63 total samples were analyzed (16 with low SCC, 19 with medium SCC, and 28 with high SCC). Sixty-one samples had lipemia (trace, n = 29; 1+, 21; 2+, 10; and 3+, 1), and 40 had hemolysis (trace, 29; 1+, 10; and 2+, 1). None of the samples were icteric.

When ELISA results were outside the reportable range for the protocol used (SCC < 0.5 or > 10.0 μg/dL for the baseline-cortisol protocol and SCC < 2.5 or > 30.0 μg/dL for the Cushing disease–suspected protocol), the involved data pairs (5 with low SCCs, 4 with medium SCCs, and 1 with a high SCC) were excluded from statistical analyses, so that only those samples for which all 3 methods yielded a measured result were included.

Results for the 2 CLAs were very highly correlated (r = 0.99; Table 1; Figure 1). When comparing results from CLA 1 with those from CLA 2 for all samples, only mild proportional bias was present (slope, 1.10; 95% CI, 1.07 to 1.13). When comparing CLA results with the ELISA results for all samples, only ELISA results produced from the more appropriate analyzer protocol (baseline-cortisol protocol results for the sample with low SCC; Cushing disease–suspected protocol results for samples with medium and high SCCs) were used. Correlations between ELISA and CLA results for all samples were very high (r > 0.90) as long as the appropriate protocol was chosen. Mean difference between CLA 1 and ELISA results (all samples) was 1.9 μg/dL (52.4 nmol/L), and mean difference between CLA 2 and ELISA results (all samples) was 1.0 μg/dL (27.6 nmol/L). However, when results for sample groups were compared, mean differences were higher as SCC increased. When results for all samples were considered, there was no statistical evidence of constant bias detected with regression analysis, and only mild proportional bias was detected when results obtained with the ELISA were compared with results of CLA 1.

Table 1—

Results of Passing-Bablok regression and Bland-Altman analysis of agreement between SCCs measured with 2 CLAs (CLA 1 and CLA 2; reference methods) and a veterinary-specific ELISA for 63 serum samples from dogs.

  Passing-Bablok regression analysis 
   y interceptSlope 
Assay comparisonNo. of sample pairsrEstimate95% CIConstant bias*Estimate95% CIProportional biasBland-Altman analysis (mean bias ± 1.96 SD)
All samples
  CLA 1 vs CLA 2630.99−0.5−0.81 to 0.18No1.101.07–1.13Yes0.9 ± 3.3
  CLA 1 vs ELISA620.96−0.29−1.63 to 0.72No1.181.05–1.34Yes1.9 ± 8.4
  CLA 2 vs ELISA620.970.17−0.59 to 0.83No1.090.980–1.18No1.0 ± 6.0
Samples with low SCC
  CLA 1 vs CLA 2160.980.00−0.38 to 0.38No1.000.9–1.13No−0.01 ± 0.6
  CLA 1 vs ELISA (BP)160.830.69−0.94 to 1.43No0.990.67–1.43No0.66 ± 1.5
  CLA 2 vs ELISA (BP)160.860.820.32 to 1.37Yes0.960.69–1.23No0.67 ± 1.4
  CLA 1 vs ELISA (CDSP)130.340.96−1.46 to 3.15No0.850.33–1.44No−0.1 ± 4.1
  CLA 2 vs ELISA (CDSP)130.410.56−2.13 to 2.86No0.980.33–1.7No−0.1 ± 4.2
Samples with medium SCC
  CLA 1 vs CLA 2190.980.53−0.44 to 1.6No0.950.85–1.08No0.16 ± 1.4
  CLA 1 vs ELISA (BP)150.84−3.3−11.6 to −0.2Yes1.71.19–2.78Yes1.6 ± 3.6
  CLA 2 vs ELISA (BP)150.79−3.86−17.4 to −0.86Yes1.81.26–3.52Yes1.4 ± 3.9
  CLA 1 vs ELISA (CDSP)190.822.921.32 to 4.8Yes0.720.52–0.95Yes0.1 ± 4.4
  CLA 2 vs ELISA (CDSP)190.872.670.34 to 3.52Yes0.760.61–1.04No0.0 ± 3.9
Samples with high SCC
  CLA 1 vs CLA 2280.99−2.5−4.18 to −1.07Yes1.211.13–1.3Yes2.0 ± 3.9
  CLA 1 vs ELISA (CDSP)270.76−19.9−38.8 to −7.36Yes2.231.58–3.26Yes3.9 ± 11
  CLA 2 vs ELISA (CDSP)270.81−14.0−27.2 to −2.81Yes1.841.25–2.56Yes2.0 ± 8.1

Constant bias was present if the 95% CI for the y intercept of the regression line did not include 0.

Proportional bias was present if the 95% CI for the slope of the regression line did not include 1.

BP = Baseline-cortisol protocol for the ELISA analyzer (reported detection range for SCC, 0.5 to 10.0 μg/dL) was used. CDSP = Cushing disease–suspected protocol for the ELISA analyzer (reported detection range for SCC, 2.5 to > 30.0 μg/dL) was used. High SCC = SCC of 18.0 to 30.0 μg/dL. Low SCC = SCC of 0.5 to 6.0 μg/dL. Medium SCC = SCC of 7.0 to 17.0 μg/dL.

Figure 1—
Figure 1—

Passing-Bablok regression (A, C, and E) and Bland-Altman (B, D, and F) plots comparing results of 2 CLAs (CLA 1 and CLA 2) and a veterinary-specific ELISA for measuring cortisol concentrations in 63 serum samples from dogs. A, C, and E—The solid line represents the linear regression line, the dashed lines indicate the 95% CI for the regression line, and the dotted line represents perfect agreement between methods (ie, a regression line slope of 1). B, D, and F—The solid line indicates the mean bias, and the dotted lines represent the 95% limits of agreement.

Citation: Journal of the American Veterinary Medical Association 253, 12; 10.2460/javma.253.12.1580

Enzyme-linked immunosorbent assay test kit failure rate

A failure in the development phase of the ELISA, which led to an error result, occurred with 35 of the total 305 (11.5%) ELISA kits used. In these instances, the sample analysis was repeated (1 to 3 times) until a numerical value was reported.

Error grid analysis

Of the 63 results evaluated, 47 (75%) were in zone A, indicating clinical agreement between ELISA results and CLA results (Figure 2). However, clinically important disagreement occurred with 16 (zone B [clinical disagreement], n = 13; zone C [clinical misdiagnosis], 3) of 63 (25%) of samples, with the greatest clinical disagreement noted when SCCs were very high or very low, which are considered critical for clinical diagnosis.

Figure 2—
Figure 2—

Error grid depicting clinical agreement between SCCs obtained with reference methods (mean of values for CLA 1 and CLA 2) and a veterinary-specific ELISA for 63 serum samples from dogs. Decision thresholds were those recommended by the ELISA manufacturer for interpretation of SCC following ACTH stimulation testing. Zone A = Zone of clinical agreement (n = 47/63 [75%]). Zone B = Zone of clinical disagreement leading to delayed diagnosis or inappropriately conclusive or inconclusive results (13/63 [21%]). Zone C = Zone of clinical misdiagnosis or inappropriate decision (3/63 [5%]).

Citation: Journal of the American Veterinary Medical Association 253, 12; 10.2460/javma.253.12.1580

Discussion

Findings from method comparison suggested that for all samples combined, SCCs obtained with the commercial ELISA highly correlated with SCCs obtained with the 2 validated CLAs when the appropriate protocol (baseline-cortisol vs Cushing disease–suspected) was selected on the ELISA analyzer. However, constant and proportional biases between the ELISA and CLAs were present at all concentrations, with ELISA results typically lower (ie, negative constant bias) than CLA results. Clinically important disagreement between ELISA results and CLA results was found for 16 of 63 (25%) samples, with the greatest clinical disagreement noted when SCCs were very high or very low, which are considered critical for clinical diagnosis. Finally, an ELISA error rate of 11.5% (35/305), despite the procedure having been run by well-qualified personnel, required repeated sample analysis in the present study.

Precision performance claims by the manufacturer of the ELISA used in the present study were not available for comparison with our precision data. However, the CLA methods had CVs ≤ 8.3% across a range of cortisol concentrations. Formal analytic quality requirements for endocrine testing in veterinary medicine did not exist at the time of the present study, and biological variation data for measurement of SCC in dogs have not been reported. In human medicine, the performance requirement for TE of a cortisol assay, stipulated by the Clinical Laboratory Improvement Amendment, is “target ± 25%,” indicating that cortisol assays must provide values within 25% of the target concentration, which can be used as a starting point for discussing analytic performance of veterinary cortisol assays.12 If, for the sake of discussion, one assumed that imprecision (ie, random error) was the only source of analytic error for cortisol assays (ie, one assumed that there was no constant or proportional bias), then imprecision should be < 25% on the basis of the Clinical Laboratory Improvement Amendment criterion. However, imprecision is only 1 component of TE, which is often calculated as the percentage bias plus twice the CV (TE = %bias + 2CV). Thus, for an assay without any bias, TE would be twice the CV (TE = 2CV), meaning that the CV ideally should be < 12.5%. Given this, the within-laboratory CVs for pooled serum samples with medium and low SCCs (28.9% and 21.4%, respectively) in the present study were higher than optimal, whereas the CV for the pooled sample with a high SCC (13.0%) could be considered acceptable.

Similar to the case for both CLAs, precision of the ELISA method was greatest (ie, lowest CV) at high SCCs. Reasons why the ELISA had greater imprecision at medium and low SCCs were unknown, and more precision data spanning a wider range of SCCs and including a larger number of repetitions would be needed to better characterize the ELISA precision profile (eg, to characterize any mathematical relationship between imprecision and SCC for this assay). A clinical implication of the ELISA's overall imprecision in the present study was that it raised concern that patients' results could be misclassified.

In the present study, there were high overall correlations between results of the 2 CLAs as well as between results of the ELISA and each CLA, given the more appropriate protocol was selected for interpretation by the ELISA analyzer. However, when a protocol designed for measurement of high SCCs (ie, Cushing disease–suspected protocol) was used to analyze samples with relatively low SCCs, the correlation was low. The reasons for this low correlation were unknown. Potential factors included that the Cushing disease–suspected protocol required a smaller sample volume (25 μL) than the baseline-cortisol protocol (100 μL), and although the manufacturer's instructions were followed for all measurements, the study was not designed to investigate any impact of the pipetting technique or sample volume variation on results. This finding illustrated the importance of choosing the ELISA analyzer protocol on the basis of the clinical situation at hand (eg, anticipated SCC or type of testing being performed). Notably, when the ELISA is used for monitoring dogs being treated for hyperadrenocorticism, it is possible that a less-than-optimal protocol could be chosen because the operator must choose a protocol on the basis of expected SCC and could anticipate the expected concentration incorrectly.

Importantly, a high correlation between results of the 2 assays means that the results trend in the same direction without necessarily implying that there is perfect agreement between the assays. In the present study, there was good overall analytic agreement between results of the 2 CLAs and ELISA results; however, there was not necessarily high clinical agreement between the ELISA and the CLAs on the basis of the error grid analysis. Given that CLA 1 and CLA 2 used the same reagents and antibody, the cause of the small proportional bias identified between them at higher SCCs was unknown and could have reflected differences in instrument function or local operating conditions. For comparisons of the ELISA with each CLA, constant or proportional bias, alone or in combination, was present for all SCC groups. Bias was greatest at medium and high SCCs, and the ELISA SCC measurements were lower (had negative bias), compared with measurements obtained by the CLAs. Identification of bias between the ELISA and CLAs was not surprising, given the different methodologies. Furthermore, this finding was similar to that reported in a previous study13 that evaluated the use of a human ELISA for measurement of cortisol concentrations in plasma from dogs and showed that the ELISA underestimated cortisol concentrations, compared with expected concentrations. The bias detected in the present study suggested that a clinician interpreting SCCs obtained with the ELISA method should expect the results to be, on average, lower than what would have been reported if concentrations had been measured with a CLA. Additionally, the presence of bias indicated that instrument-specific decision thresholds could be necessary.

Errors that occurred during color development with the ELISA could have been a result of improper storage of the ELISA test kits or reagents, improper temperature of the ELISA test kit at the time of use, inadequate or excessive incubation time, or unknown variables affecting serum sample characteristics. In the present study, however, all testing and storage conditions were in accordance with manufacturer's recommendations and were verified by a member of the manufacturer's technical support staff. Moreover, all ELISA tests were performed in the same room and by the same trained operator, minimizing environmental effects and preanalytic errors. It was possible that the ELISA error rate observed in the present study resulted from an inherent factor affecting the matrix of only the lot of test kits used; however, the error rate experienced in the present study was similar to the error rate that occurred in a pilot study (unpublished data) by the authors when test kits from a different lot number were used. Findings from the present study regarding the observed error rate and the error grid analysis indicated that clinical use of the ELISA for measuring SCC in dogs would necessitate repeated or follow-up confirmatory testing in some cases. At the time of the present study, the manufacturer offered replacement SCC ELISA kits at no charge when an error occurred. The authors believe that the financial and time costs of repeated testing must be considered when choosing the ELISA to measure SCC in dogs.

In the present study, we used error grid analysis to determine whether the identified bias would have affected clinical decision-making. Importantly, the error grid analysis was based solely on SCC without, for example, consideration of clinical signs, as would be the case in true clinical situations. Decision thresholds incorporated into the error grid analysis were obtained from the ELISA manufacturer but were similar to decision thresholds used at the authors' institutions. The finding of the present study, that for 47 of 63 (75%) of samples, clinical decisions made on the basis of SCC measurements obtained with the ELISA would have agreed with decisions made on the basis of measurements obtained with the CLAs, suggested that bias associated with the ELISA would have minimal clinical impact in instances when SCCs were within reference intervals or slightly above them. However, given that results of the present study indicated that clinical disagreement frequently occurred at SCCs that would be most clinically important, we conclude that abnormal SCCs reported by the ELISA should be confirmed by a reference laboratory. Although the low numbers of samples evaluated in the present study precluded strong conclusions, the rate of disagreement was concerning and suggested that equivocal or discordant results should be confirmed by a reference laboratory. The present study evaluated analytic performance of the ELISA when used to measure SCC in dogs. A potential area of further study could be to characterize the ELISA's performance when assay results are combined with other clinical criteria to diagnose hypoadrenocorticism, hyperadrenocorticism, or other conditions affecting cortisol concentration in dogs (clinical performance).

A limitation of the present study was that although samples were selected to represent a range of clinically important SCCs (0.5 to 30.0 μg/dL), only 2 samples had an SCC < 2.0 μg/dL (< 55.2 nmol/L, consistent with hypoadrenocorticism) as measured with the CLAs, and only 8 samples had an SCC > 22.0 μg/dL (> 607.0 nmol/L, consistent with hyperadrenocorticism) as measured with the CLAs. Serum cortisol concentrations considered diagnostic for hypoadrenocorticism are < 2.0 μg/dL and near the ELISA's lower limit of detection (0.5 μg/dL); thus, it was reasonable to expect that the negative bias and imprecision of the ELISA would result in limitations at low SCCs. Further study of the ELISA on samples with low SCC is needed to confirm this. The present study predominantly included samples with midrange SCCs; therefore, observed bias and error grid analysis discordance could represent minimums, compared with what could be observed in clinical practice, where the proportions of samples with very low or very high SCCs would likely be greater than those in the study reported here.

Although there was very high correlation and good overall agreement between ELISA results and results of the 2 CLAs in the present study, our findings indicated that the ELISA had imprecision and that there was clinical discordance in SCC measurements for 16 of the 63 (25%) samples on the basis of error grid analysis. The choice of analytic protocol for the ELISA affected the degree of agreement between the ELISA and CLAs in the present study, and the presence of bias between SCC measurement methods illustrated the importance of having method-specific decision thresholds. Clearly, results of SCC measurements should not be interpreted in a vacuum, and veterinarians obtaining equivocal results (or results that seem discordant with clinical findings) should consider confirmatory SCC testing through a reference laboratory. As new cortisol assays are developed for clinical use, testing sites should independently verify analytic performance of the assays to ensure that instruments and methods being used meet manufacturer performance claims and are suitable for their intended clinical purposes.

Acknowledgments

Funded by grants provided by the Society for Comparative Endocrinology and the University of Tennessee's Companion Animal Fund. The Idexx SNAPshotDX Analyzer used in the study was provided by the Regional Institute for Veterinary Emergencies and Referrals of Chattanooga, Tenn.

Presented as an oral abstract at the American College of Veterinary Internal Medicine Forum, Washington, DC, 2017.

The authors declare that there were no conflicts of interest.

The authors thank Julie Fields, Deanne Gibbs, Ivan Haworth, and Jessie Cagle from the Department of Biomedical and Diagnostic Sciences at the University of Tennessee, and Susan Beyerlein, Laboratory Manager of the Department of Pathobiology and Diagnostic Investigation at the Michigan State University College of Veterinary Medicine, for their assistance in data collection.

ABBREVIATIONS

CI

Confidence interval

CLA

Chemiluminescence assay

CV

Coefficient of variation

RIA

Radioimmunoassay

SCC

Serum cortisol concentration

sr

Repeatability SD

sl

Within-laboratory SD

TE

Total analytic error

Footnotes

a.

Coat-a-Count Cortisol, Siemens Medical Solutions USA Inc, Malvern, Pa.

b.

SNAP Cortisol Test Kit, Idexx Laboratories Inc, Westbrook, Me.

c.

SNAPshot DX Analyzer, Idexx Laboratories Inc, Westbrook, Me.

d.

Immulite 1000 Immunoassay Analyzer, Siemens Healthcare USA Inc, Malvern, Pa.

e.

Canine serum-based control K9CON levels 1,2 (part No. 10385584: lot No. 36; level 1, kit lot No. 403-406; level 2, kit lot No. 403-406) for Immulite 1000, Siemens Medical Solutions USA Inc, Malvern, Pa.

f.

Immulite 2000XPi Immunoassay Analyzer, Siemens Healthcare USA Inc, Malvern, Pa.

g.

Lyphochek Immunoassay Plus control levels 1, 2 and 3 (lot No. 40310) for Immulite 2000 XPi, Bio-Rad Laboratories Inc, Hercules, Calif.

h.

MedCalc statistical software, version 15.11.0, MedCalc Software bvba, Ostend, Belgium.

References

  • 1. Melián C, Peterson ME. Diagnosis and treatment of naturally occurring hyperadrenocorticism in 42 dogs. J Small Am Anim Pract 1996;37:268275.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 2. Behrend EN, Kemppainen RJ. Diagnosis of canine hyperadrenocorticism. Vet Clin North Am Small Anim Pract 2001;31:9851003.

  • 3. Reimers TJ, Cowan RG, Davidson HP, et al. Validation of radioimmunoassays for triiodothyronine, thyroxine, and hydrocortisone (cortisol) in canine, feline, and equine sera. Am J Vet Res 1981;42:20162021.

    • Search Google Scholar
    • Export Citation
  • 4. Russell NJ, Foster S, Clark P, et al. Comparison of radioimmunoassay and chemiluminescent assay methods to estimate canine blood cortisol concentrations. Aust Vet J 2007;85:487494.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 5. Reimers TJ, Salerno VJ, Lamb SV. Validation and application of solid-phase chemiluminescent immunoassays for diagnosis of endocrine diseases in animals. Comp Haematol Int 1996;6:170175.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 6. Ramsey I, Herrtage M. Laboratory evaluation of adrenal diseases. In: Villier E, Ristic J, eds. BSAVA manual of canine and feline clinical pathology. 3rd ed. Hoboken, NJ: John Wiley & Sons, 2016;353372.

    • Search Google Scholar
    • Export Citation
  • 7. Flatland B, Freeman KP, Friedrichs KR, et al. ASVCP quality assurance guidelines: control of general analytical factors in veterinary laboratories. Vet Clin Pathol 2010;39:264277.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 8. Smith MB, Chan YW, Dolci A, et al. Hemolysis, icterus, and lipemia/turbidity indices and indicators of interference in clinical laboratory analysis; approved guideline. CLSI document C-56A. Wayne, Pa: Clinical and Laboratory Standards Institute, 2012.

    • Search Google Scholar
    • Export Citation
  • 9. Clinical and Laboratory Standards Institute. User verification of performance for precision and trueness: approved guideline. 2nd ed. CLSI document EP15–A2. Wayne, Pa: Clinical and Laboratory Standards Institute, 2005.

    • Search Google Scholar
    • Export Citation
  • 10. SNAP Cortisol Test Kit [package insert]. Westbrook, Me: Idexx Laboratories Inc, 2015.

  • 11. Mukaka MM. Statistics corner: a guide to appropriate use of correlation coefficient in medical research. Malawi Med J 2012;24:6971.

    • Search Google Scholar
    • Export Citation
  • 12. Department of Health and Human Services. Medicare, Medicaid, and CLIA programs; regulations implementing the clinical laboratory improvements amendments of 1988 (CLIA). Final rule. Fed Regist 1992;57:70027186.

    • Search Google Scholar
    • Export Citation
  • 13. Ginel PJ, Perez-Rico A, Moreno P, et al. Validation of a commercially available enzyme-linked immunosorbent assay (ELISA) for the determination of cortisol in canine plasma samples. Vet Res Commun 1998;22:179185.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Figure 1—

    Passing-Bablok regression (A, C, and E) and Bland-Altman (B, D, and F) plots comparing results of 2 CLAs (CLA 1 and CLA 2) and a veterinary-specific ELISA for measuring cortisol concentrations in 63 serum samples from dogs. A, C, and E—The solid line represents the linear regression line, the dashed lines indicate the 95% CI for the regression line, and the dotted line represents perfect agreement between methods (ie, a regression line slope of 1). B, D, and F—The solid line indicates the mean bias, and the dotted lines represent the 95% limits of agreement.

  • Figure 2—

    Error grid depicting clinical agreement between SCCs obtained with reference methods (mean of values for CLA 1 and CLA 2) and a veterinary-specific ELISA for 63 serum samples from dogs. Decision thresholds were those recommended by the ELISA manufacturer for interpretation of SCC following ACTH stimulation testing. Zone A = Zone of clinical agreement (n = 47/63 [75%]). Zone B = Zone of clinical disagreement leading to delayed diagnosis or inappropriately conclusive or inconclusive results (13/63 [21%]). Zone C = Zone of clinical misdiagnosis or inappropriate decision (3/63 [5%]).

  • 1. Melián C, Peterson ME. Diagnosis and treatment of naturally occurring hyperadrenocorticism in 42 dogs. J Small Am Anim Pract 1996;37:268275.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 2. Behrend EN, Kemppainen RJ. Diagnosis of canine hyperadrenocorticism. Vet Clin North Am Small Anim Pract 2001;31:9851003.

  • 3. Reimers TJ, Cowan RG, Davidson HP, et al. Validation of radioimmunoassays for triiodothyronine, thyroxine, and hydrocortisone (cortisol) in canine, feline, and equine sera. Am J Vet Res 1981;42:20162021.

    • Search Google Scholar
    • Export Citation
  • 4. Russell NJ, Foster S, Clark P, et al. Comparison of radioimmunoassay and chemiluminescent assay methods to estimate canine blood cortisol concentrations. Aust Vet J 2007;85:487494.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 5. Reimers TJ, Salerno VJ, Lamb SV. Validation and application of solid-phase chemiluminescent immunoassays for diagnosis of endocrine diseases in animals. Comp Haematol Int 1996;6:170175.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 6. Ramsey I, Herrtage M. Laboratory evaluation of adrenal diseases. In: Villier E, Ristic J, eds. BSAVA manual of canine and feline clinical pathology. 3rd ed. Hoboken, NJ: John Wiley & Sons, 2016;353372.

    • Search Google Scholar
    • Export Citation
  • 7. Flatland B, Freeman KP, Friedrichs KR, et al. ASVCP quality assurance guidelines: control of general analytical factors in veterinary laboratories. Vet Clin Pathol 2010;39:264277.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 8. Smith MB, Chan YW, Dolci A, et al. Hemolysis, icterus, and lipemia/turbidity indices and indicators of interference in clinical laboratory analysis; approved guideline. CLSI document C-56A. Wayne, Pa: Clinical and Laboratory Standards Institute, 2012.

    • Search Google Scholar
    • Export Citation
  • 9. Clinical and Laboratory Standards Institute. User verification of performance for precision and trueness: approved guideline. 2nd ed. CLSI document EP15–A2. Wayne, Pa: Clinical and Laboratory Standards Institute, 2005.

    • Search Google Scholar
    • Export Citation
  • 10. SNAP Cortisol Test Kit [package insert]. Westbrook, Me: Idexx Laboratories Inc, 2015.

  • 11. Mukaka MM. Statistics corner: a guide to appropriate use of correlation coefficient in medical research. Malawi Med J 2012;24:6971.

    • Search Google Scholar
    • Export Citation
  • 12. Department of Health and Human Services. Medicare, Medicaid, and CLIA programs; regulations implementing the clinical laboratory improvements amendments of 1988 (CLIA). Final rule. Fed Regist 1992;57:70027186.

    • Search Google Scholar
    • Export Citation
  • 13. Ginel PJ, Perez-Rico A, Moreno P, et al. Validation of a commercially available enzyme-linked immunosorbent assay (ELISA) for the determination of cortisol in canine plasma samples. Vet Res Commun 1998;22:179185.

    • Crossref
    • Search Google Scholar
    • Export Citation

Advertisement