On-site communication skills education increases appointment-specific client satisfaction in four companion animal practices in Texas

Natasha Janke Department of Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Colorado State University, Fort Collins, CO

Search for other papers by Natasha Janke in
Current site
Google Scholar
PubMed
Close
 PhD
,
Jane R. Shaw Department of Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Colorado State University, Fort Collins, CO

Search for other papers by Jane R. Shaw in
Current site
Google Scholar
PubMed
Close
 DVM, PhD
, and
Jason B. Coe Department of Population Medicine, Ontario Veterinary College, University of Guelph, Guelph, ON, Canada

Search for other papers by Jason B. Coe in
Current site
Google Scholar
PubMed
Close
 DVM, PhD

Abstract

Objective

To assess the impact of a 15-month, in-practice communication skills intervention on client visit satisfaction.

Sample

4 purposely selected veterinary clinics, 9 veterinarians, and 930 appointment-specific client satisfaction surveys.

Procedures

The study was designed as a multipractice, preintervention-postintervention study. Practice teams participated in a 15-month communication skills intervention that included interactive group workshops and one-on-one communication coaching with the entire practice. Client Satisfaction Questionnaires were completed 3 months before (mean, 55 questionnaires/veterinarian) and after (mean, 48 questionnaires/veterinarian) the intervention. The impact of the intervention and factors associated with client visit satisfaction were evaluated with mixed logistic regression.

Results

Client satisfaction scores were dichotomized owing to extreme negative skewness. In total, 57% (284/496) of clients were completely satisfied on pretest questionnaires, and 67% (290/434) were completely satisfied on postintervention questionnaires. The odds of clients being completely satisfied were significantly higher after the communication skills intervention (OR = 1.56; P = .002). When controlling for the intervention, predictors of client visit satisfaction included appointment type (odds of being completely satisfied were significantly higher for recheck appointments than for preventative care appointments [OR = 1.71; P = .02] and health problem appointments [OR = 1.99; P = .003]) and duration of the veterinarian-client relationship (probability increased by 0.52 with each 1-year increase in relationship duration; P = .008).

Clinical Relevance

Results suggested that the client-centered, skills-based communication intervention resulted in increased client visit satisfaction. Findings contribute to current evidence that client-centered communication and duration of the veterinarian-client relationship promote client satisfaction during veterinary visits.

Abstract

Objective

To assess the impact of a 15-month, in-practice communication skills intervention on client visit satisfaction.

Sample

4 purposely selected veterinary clinics, 9 veterinarians, and 930 appointment-specific client satisfaction surveys.

Procedures

The study was designed as a multipractice, preintervention-postintervention study. Practice teams participated in a 15-month communication skills intervention that included interactive group workshops and one-on-one communication coaching with the entire practice. Client Satisfaction Questionnaires were completed 3 months before (mean, 55 questionnaires/veterinarian) and after (mean, 48 questionnaires/veterinarian) the intervention. The impact of the intervention and factors associated with client visit satisfaction were evaluated with mixed logistic regression.

Results

Client satisfaction scores were dichotomized owing to extreme negative skewness. In total, 57% (284/496) of clients were completely satisfied on pretest questionnaires, and 67% (290/434) were completely satisfied on postintervention questionnaires. The odds of clients being completely satisfied were significantly higher after the communication skills intervention (OR = 1.56; P = .002). When controlling for the intervention, predictors of client visit satisfaction included appointment type (odds of being completely satisfied were significantly higher for recheck appointments than for preventative care appointments [OR = 1.71; P = .02] and health problem appointments [OR = 1.99; P = .003]) and duration of the veterinarian-client relationship (probability increased by 0.52 with each 1-year increase in relationship duration; P = .008).

Clinical Relevance

Results suggested that the client-centered, skills-based communication intervention resulted in increased client visit satisfaction. Findings contribute to current evidence that client-centered communication and duration of the veterinarian-client relationship promote client satisfaction during veterinary visits.

Introduction

Appointment-specific client satisfaction has evolved over the past decades alongside veterinary client expectations. In the early 2000s, there was an increased recognition of the human-animal bond14 and the impact of this attachment on clients’ behaviors when seeking veterinary care for their pets.5 Findings from client focus groups and semistructured interviews indicate that the veterinarian-client interaction and relationship are also important facets for seeking veterinary care.68

Clients want their veterinarian to listen to their concerns and observations regarding their pet, use clear, easily understood language when providing information, treat them as a partner, and provide sufficient information to allow them to make fully informed decisions.68 Clients also prefer client education tailored to their prior knowledge and understanding of a topic.8,9 These findings emphasize the importance of reciprocal communication with clients in a manner that meets their needs.

Previous evidence suggests that veterinarian-client communication is associated with client satisfaction10 and adherence to veterinarian recommendations.11 In a study that assessed veterinarians’ medical dialogue in 64 veterinary consultations, McArthur and Fitzgerald10 found that client satisfaction, measured globally, was significantly higher for appointments during which the veterinarian expressed empathy toward the client. Kanji et al11 found that displaying a sympathetic and empathetic tone and adopting an unhurried and unrushed manner increased client adherence.11 Although a causal relationship between client satisfaction and adherence has yet to be established, clients who adhered to dentistry and surgery recommendations were significantly more satisfied than nonadhering clients.11

Findings vary regarding the impact of communication skills education on veterinary client satisfaction.1214 McArthur and Fitzgerald14 measured the effects of a 6.5-hour communication skills intervention with a pretest-posttest single-group study design. Although client satisfaction did not increase, clients’ intent to adhere to their veterinarian’s recommendation improved after the intervention. Cornell et al12 assessed the impact of a 2-day training session followed by biweekly, in-practice training sessions for 3 months. Client satisfaction was high before the intervention and did not differ significantly after it; however, veterinary team members’ communication skills confidence and their ability to initiate conversations with clients regarding the value of veterinary products and services were improved following the intervention.12 After a 6-month, in-practice communication intervention, Shaw et al13 found that client visit satisfaction was enhanced in relation to clients’ perceived involvement in the appointment and their veterinarian’s interest in their opinion. Understanding predictors of client satisfaction provides valuable evidence for future training of veterinary professionals.

The present study represents part of a larger study of the impact of a 15-month communication skills training intervention. The purpose of the study reported here was to evaluate the impact of an in-practice communication skills intervention on client visit satisfaction, using the Client Satisfaction Questionnaire (CSQ), a validated and reliable instrument for measuring appointment-specific satisfaction.15 The research objectives were to evaluate client visit satisfaction before and after communication skills training (ie, the intervention) and identify predictor variables associated with client visit satisfaction.

Materials and Methods

Study design

A preintervention-postintervention study was conducted at 4 multidoctor companion animal veterinary practices that were part of a group practice consisting of veterinary clinics surrounding Austin, Texas. In-practice communication training was conducted between April 2018 and June 2019. Survey data and video recordings were collected 3 months before and after the intervention period. The Human Subjects Research Committee of the Colorado State University Research Integrity and Compliance Review Office approved the research protocol. This study was also approved by the University of Guelph Research Ethics Board (REB#16-12-606).

Recruitment

Veterinary practices—A convenience sample of all practices that were in operation by a single practice group known to the principal investigator (JRS) at the time of pretest data collection was enrolled in the study.

Veterinary team members—Participants were recruited at the level of the 4 practices, with all veterinarians and veterinary team members employed by these 4 practices invited to participate in the study. Veterinary team members provided online consent to participate at the beginning of the study period and on a rolling basis for employees hired during the study period.

Clients—A convenience sample of clients (mean, 52 clients/participating veterinarian before and after the intervention) was enlisted in person, at the time of arrival for their appointments, by a veterinary team member. Clients who agreed to participate were asked to complete the CSQ. Online informed consent was obtained from these participating clients at the beginning of the client visit satisfaction survey.

Video-recorded appointments—In addition, for each veterinarian included in the study, 16 clients (8 before and 8 after the intervention) were solicited for permission to video record their appointment with a standard, tripod-mounted digital video camera that was set up in a corner of the examination room. Written in-person informed consent was obtained from these client participants by an assigned veterinary team member trained to obtain client consent, at the time of arrival for their appointments, and online consent was obtained at the beginning of the client visit satisfaction survey.

Demographic data

Demographic data were collected from all participants with an electronic survey completed during the pre- and postintervention data collection periods. For veterinary practices, this information consisted of the type and location of the practice, number of veterinarians employed, and number of veterinary team members employed. For veterinarians, this information consisted of gender, age, job title, years in current position, and years since graduation from veterinary school. For veterinary team members, this information consisted of gender, age, job title, years in current position, and number of years in the profession. For clients, this information consisted of gender, age, highest level of education achieved, household income, number of veterinary visits per year, and duration of veterinarian-client relationship (years). Clients were also asked to report the age, species, and sex of their pets.

Appointment type

Appointment type was categorized as preventative care, initial health problem, recheck, or emergency visit and was identified by the client at the beginning of the visit-specific client satisfaction survey. Preventative care appointments were encounters with a presumably healthy pet that was presented for a routine examination and prophylactic treatment. Initial health problem appointments were with a patient that was presented with a health-related problem. Recheck visits were with an animal formerly seen for a health problem appointment that required a progress or follow-up appointment with a veterinarian. Emergency appointments were unscheduled visits with a dog or cat that were not previously planned on the appointment day. These definitions were not provided on the client survey; however, clients were able to clarify the appointment type with the veterinary professional administering the survey.

Client visit satisfaction

Client satisfaction was measured with the 15-item CSQ developed by Coe et al15 to measure visit-specific client satisfaction with companion animal visits. The CSQ exhibits high internal consistency (G coefficient = 0.96) and construct validity (r = 0.9).15 Global client satisfaction was measured with a single item (“Based on today’s experience, please rate your overall experience today with this veterinarian”) rated on a scale of 0 (poor) to 100 (could not be better). Clients completed the CSQ after their visit with the veterinarian.

The study aimed to obtain a mean of 60 completed CSQs/veterinarian during data collection periods 3 months before and 3 months after the intervention, for a total goal of 120 CSQs/veterinarian, with 16 of the appointments related to the 120 CSQs (8 before and 8 after the intervention) video recorded. The sample size was based on the number of CSQs previously identified to reliably differentiate between veterinarian appointment–specific satisfaction.15 In contrast to the original CSQ, which measured items on a 6-point Likert scale, the items were measured on a visual analog scale from 0 (poor) to 100 (could not be better) in an attempt to capture greater variation in participating clients’ satisfaction and to create a continuous variable for data analyses (ie, modified CSQ). To test the impact of using a visual analog scale, internal consistency and construct validity were reassessed.15

Intervention

The communication skills intervention consisted of 7 visits to each practice. The duration of each visit was 4 hours for the 2 smaller practices and 8 hours for the 2 larger practices owing to the total number of practice team members. Each session entailed previsit work, an interactive evidence- and skills-based communication workshop with the entire practice team that used the Calgary-Cambridge Guide (CCG) communication framework as its foundation,16 and one-on-one communication coaching with veterinary team members.

Over the 7 visits, the curriculum progressed chronologically through the clinical interview structure (initiating the interview, history gathering, building a relationship, explaining and planning, providing structure, and closing the interview) and featured 20 key communication skills from the CCG to accomplish these clinical tasks (Appendix 1). The training sessions were standardized between practices, were conducted by a single member of the research team (JRS) who had extensive experience in veterinarian-client-patient interactions and skill-based communication education, and used a flipped-classroom design.17 Prior to each session, veterinary team members completed readings focusing on that session’s CCG interview tasks and associated communication skills to enhance foundational communication skills knowledge. The subsequent workshop with all team members lasted approximately 1.5 hours and focused on implementation and application of the communication skills. These interactive workshops were offered in the form of a lunch-and-learn and included role playing, small group discussions, and brief communication skill drills.

The remaining time at each practice involved one-on-one communication skills coaching with participating veterinary team members led by the same individual (JRS). Coaching took place while participating veterinary team members interacted with clients during scheduled appointments. Verbal in-the-moment feedback was provided to each participant during the observation period highlighting communication skills of strength and areas for growth. Participants also received descriptive and specific written feedback following each interaction. Each team member was observed at least once during each visit and as many as 2 to 5 times during a visit.

Between training sessions, hospital managers led 30-minute communication rounds with the entire team every other week throughout the 15-month intervention period. These sessions represented an opportunity to present case scenarios of client or colleague interactions and identify successes, challenges, or opportunities to implement the communication skills. The team debriefed what worked well and what might have been done differently and offered suggestions for next steps. A scheduled monthly teleconference was conducted between the on-site communication coach, a researcher (JRS), and the veterinary group leadership and hospital managers from all 4 hospitals to assess progress and brainstorm challenges.

Statistical analysis

Descriptive statistics that were calculated included frequency distributions for categorical variables and mean, median, range, and SD for continuous variables. An overall mean satisfaction score was calculated by averaging scores across the total number of item responses.15 The distribution of continuous variables was tested for normality with the Shapiro-Wilk test. Pre- and postintervention client and appointment demographics were assessed for differences with a χ2 test for categorical variables, a t test for continuous variables with a normal distribution, or a Mann-Whitney U test for continuous variables that were not normally distributed.

Differences between modified-CSQ scores from clients who had their appointment video recorded and those who did not were assessed with the Mann-Whitney U test to examine the potential impact of video recording appointments on veterinarians’ behavior in the examination room, ultimately affecting CSQ scores. There was no significant difference between modified-CSQ scores from clients whose appointments were video recorded and those that were not (preintervention, P = .55; postintervention, P = .24). Therefore, all modified-CSQ scores were considered for further analyses. Subsequently, the Mann-Whitney U test was used to compare mean modified-CSQ scores for participants with complete data and modified-CSQ scores for participants with ≥ 1 missing value. A significant difference was identified between groups (preintervention, P = .018 [n = 581]; postintervention, P = .001 [621); therefore, only complete scores were used in subsequent analyses.

The psychometric properties of the modified CSQ were evaluated by calculating the Cronbach α to assess internal consistency. Construct validity was assessed with the Spearman rank test to assess the relationship between a client’s mean modified-CSQ score and their global rating of visit satisfaction.

Overall mean modified-CSQ scores were transformed and reassessed for normality, including the degree of skewness and kurtosis. Owing to the extreme negative skewness of the data, overall mean modified-CSQ scores were dichotomized, with 1 equal to a mean score of 100 (ie, completely satisfied) and 0 equal to a mean score < 100 (ie, not completely satisfied). Once overall mean modified-CSQ scores were dichotomized, the impact of missing values was reassessed with a χ2 test, and a significant difference remained between complete and incomplete CSQs (preintervention, P = .008 [n = 581]; postintervention, P = .03 [621]). Therefore, only complete dichotomized scores were used in the final analysis. Individual CSQ item scores were assessed for differences in the frequencies of completely satisfied clients before and after the intervention with a χ2 test. No adjustments were made to P values to control for multiple comparisons.

To assess the impact of the intervention on client satisfaction, mixed logistic regression modeling was conducted with dichotomized modified-CSQ score as the outcome. To address the lack of independence between veterinarians’ repeated measures with multiple veterinarians participating from the same practice, veterinarian crossed with intervention (preintervention vs postintervention), nested within clinic, was retained as a random effect in the model.

Practice, veterinarian, client, pet, and appointment factors were assessed for unconditional associations with client satisfaction (completely satisfied vs not completely satisfied) by means of univariable mixed logistic regression. All associations with a P value < .20 were retained for testing in the main effects model. The final model was fit by means of manual backward-stepwise elimination, removing variables if the P value exceeded .05 with a type III F test, starting with the least significant. Potential confounding variables were input into the model to assess the impact on the coefficients of the main effects and were retained if this resulted in a change in the coefficients of the main effects models of ≥ 30%. Finally, all 2-way interactions between each of the predictor variables in the final main effects model were explored and retained if significant. Listwise deletion was used to handle missing data. All statistical analyses were conducted with standard software (SAS OnDemand for Academics; SAS Institute Inc).

Results

Veterinary practice demographic data

Four companion animal practices in 3 suburban and 1 rural location in Texas were included in the study. The 2 smaller practices employed 2 veterinarians and 9 to 12 veterinary team members at the start of the study, whereas the 2 larger practices employed 3 to 4 veterinarians and 12 to 19 veterinary team members.

Veterinarian demographic data

Three of 12 veterinarians were excluded from the study because they did not complete the full intervention, including preintervention and postintervention data collections. Two veterinarians departed the group practice before the postintervention data collection period, and 1 veterinarian joined the practice after the preintervention data collection period. The final analyses included 9 veterinarians from 4 practices. There were 7 females and 2 males. Five were associate veterinarians, 3 were medical directors, and 1 was a practice owner-partner. Mean ± SD time since graduation from veterinary school was 8.12 ± 3.64 years (median, 7.50 years; range, 4.0 to 14.0 years), and mean age was 35.8 ± 4.29 years (median, 35.0 years; range, 31.0 to 44.0 years). Participating veterinarians had held their current positions for a mean of 2.90 ± 3.29 years (median, 1.0 year; range, 0.2 to 10.0 years) at the start of the study.

Veterinary team member demographic data

At the start of the study, there were 33 veterinary team members, of which 10 (30%) completed the full intervention. An additional 37 veterinary team members joined the practices during the study and were present at the end of the study period. The 10 veterinary team members who completed the full intervention consisted of 5 veterinary technicians, 4 client services representatives, and 1 practice manager. Mean ± SD age of these veterinary team members was 31.2 ± 4.73 years (median, 30.0 years; range, 27 to 30 years); 9 were female and 1 was male. Veterinary team members had worked in the profession for a mean of 8 ± 6.75 years (median, 6.0 years; range, 2 to 20 years) and had held their current positions a mean of 4.45 ± 6.98 years (median, 2.0 years; range, 0.5 to 20 years) at the start of the study.

Appointment, client, and patient demographic data

Appointment, client, and patient demographic characteristics were summarized (Table 1). The only significant difference in demographic data between pre- and postintervention samples was the number of years that clients knew the participating veterinarian, which was significantly (P < .001) higher in the postintervention sample.

Table 1

Comparison of client, pet, and visit characteristics in a study assessing the impact of a 15-month, in-practice communication skills intervention on client visit satisfaction.

Variable Preintervention appointments Postintervention appointments P value
Client
 Age (y) 38.0 (17–80) 39.0 (18–76) 0.80
 Duration of veterinarian-client relationship (y) 1.0 (0–20) 1.0 (0–34) < .001
 No. of veterinary visits/y 3.0 (0–52) 3.0 (0–50) 0.82
 Gender 0.82
  Female 323 (71.9) 281 (71.7)
  Male 123 (27.4) 109 (27.8)
  Nonbinary 2 (0.5) 2 (0.5)
  Other 1 (0.2) 0 (0)
 Highest level of education 0.42
  Less than high school 1 (0.2) 2 (0.5)
  High school or equivalent 35 (7.2) 24 (5.8)
  Associate degree 32 (6.6) 30 (7.3)
  Some college 99 (20.5) 79 (19.1)
  Bachelor’s degree 210 (43.5) 200 (48.3)
  Graduate degree 89 (18.4) 59 (14.2)
  Professional degree 17 (3.5) 20 (4.8)
 Annual household income (USD) 0.26
  < $20,000 15 (3.2) 8 (2.2)
  $20,000–34,999 36 (7.7) 25 (6.7)
  $35,000–49,999 31 (6.6) 35 (9.4)
  $50,000–74,999 59 (12.8) 44 (11.6)
  $75,000–99,999 64 (13.9) 45 (11.8)
  $100,000–149,999 89 (19.0) 86 (23.1)
  $150,000–199,999 52 (11.1) 54 (14.5)
  ≥ $200,000 119 (25.6) 78 (20.7)
Patient
 Age (y) 4.0 (< 1–19) 4.0 (< 1–19) 0.45
 Sex 0.07
  Female 235 (47.5) 231 (53.4)
  Male 260 (52.5) 202 (46.6)
 Species 0.21
  Dog 419 (85.9) 352 (81.7)
  Cat 64 (13.1) 72 (16.7)
  Other 5 (1.0) 7 (1.6)
 Appointment type 0.48
  Preventative care 203 (43.0) 193 (45.7)
  Health problem 181 (38.4) 155 (36.7)
  Recheck 69 (14.6) 64 (15.2)
  Emergency 19 (4.0) 10 (2.4)

The study involved 9 veterinarians and their veterinary teams across 4 companion animal practices owned by a single practice group in Austin, Texas; Client Satisfaction Questionnaires (CSQs) were completed by clients for 496 appointments before and 434 appointments after the intervention. Data are given as median (range) or as number (%).

Client visit satisfaction

A total of 1,227 client surveys were completed by clients who interacted with participating veterinarians. Of these, 25 (2%) contained no responses to the modified-CSQ items and were excluded. In addition, 171 (14%) surveys included in the data set represented appointments that were video-recorded interaction. However, a technical malfunction occurred during one of these appointments.

In total, 930 (76%) CSQs with no missing data were used in the final analyses. The question that was most left unanswered by clients was item 10 (“The veterinarian’s discussion of cost with you”; n = 228 [18.6%]). Clients were able to select “not applicable” for each item of the CSQ. The output data did not differentiate between unanswered items and a selection of “not applicable”; therefore, both were considered missing values in the analysis and were not further explored.

Internal consistency of the modified CSQ that incorporated the visual analog scale was high (Cronbach α = 0.95 [n = 930]). There was a significant, moderate positive correlation between overall mean modified-CSQ score and global satisfaction score (r = 0.53; P < .001 [n = 929]). The overall mean preintervention modified-CSQ score was 97.95 ± 5.97 (median, 100; range, 42.53 to 100.00), and the overall mean postintervention score was 98.79 ± 3.49 (median, 100; range, 65.80 to 100.00; Table 2).

Table 2

Global satisfaction rating, overall modified-CSQ score, and individual item scores for the 496 preintervention and 434 postintervention appointments in Table 1.

Preintervention appointments Postintervention appointments
Variable Mean ± SD Range Mean ± SD Range
Global satisfaction rating 98.64 ± 5.59 35.00–100.00 99.52 ± 2.53 73.00–100.00
Overall modified-CSQ score 97.95 ± 5.97 42.53–100.00 98.79 ± 3.49 65.80–100.00
Item 1: Attention to pet 98.59 ± 5.70 33.00–100.00 99.48 ± 2.76 64.00–100.00
Item 2: Veterinarian understood reason for visit 99.32 ± 4.04 41.00–100.00 99.59 ± 2.11 76.00–100.00
Item 3: Veterinarian’s confidence 98.80 ± 5.84 22.00–100.00 99.26 ± 5.40 0.00–100.00
Item 4: Involved you in entire appointment 98.69 ± 5.47 51.00–100.00 99.50 ± 2.90 60.00–100.00
Item 5: Examination of pet 98.78 ± 5.05 43.00–100.00 99.51 ± 2.82 60.00–100.00
Item 6: Explained treatment and procedures 98.67 ± 5.65 34.00–100.00 99.38 ± 3.65 50.00–100.00
Item 7: You understood the costs 95.54 ± 14.24 0.00–100.00 96.65 ± 11.43 0.00–100.00
Item 8: Involved you in decisions 98.95 ± 4.72 50.00–100.00 99.25 ± 3.72 50.00–100.00
Item 9: Discussed options with you 97.89 ± 7.70 33.00–100.00 99.01 ± 4.67 52.00–100.00
Item 10: Discussed cost with you 93.12 ± 18.27 0.00–100.00 93.91 ± 17.66 0.00–100.00
Item 11: Interested in your opinion 98.24 ± 7.43 14.00–100.00 99.32 ± 3.70 60.00–100.00
Item 12: Information you received 98.42 ± 7.19 22.00–100.00 99.13 ± 4.43 40.00–100.00
Item 13: Addressed your concerns 98.93 ± 4.76 40.00–100.00 99.53 ± 2.78 60.00–100.00
Item 14: Recognized role of pet in your life 97.32 ± 11.36 0.00–100.00 98.95 ± 5.74 30.00–100.00
Item 15: Amount of time spent with you 98.01 ± 8.69 18.00–100.00 99.32 ± 3.48 65.00–100.00

Potential responses ranged from 0 (poor) to 100 (could not be better). For all variables, the median score was 100.

After modified-CSQ scores were dichotomized, 57% (284/496) of clients were completely satisfied during the preintervention period, and 67% (290/434) of clients were completely satisfied during the postintervention period. For 12 of 15 dichotomized CSQ items, the proportion of completely satisfied clients was significantly higher during the postintervention period than during the preintervention period (Table 3).

Table 3

Number (percentage) of clients who indicated they were completely satisfied (ie, modified-CSQ score = 100) for the 496 preintervention and 434 postintervention appointments in Table 1.

Variable Preintervention appointments Postintervention appointments P value
Item 1: Attention to pet 428 (86) 403 (93) 0.001
Item 2: Veterinarian understood reason for visit 455 (92) 409 (94) 0.14
Item 3: Veterinarian’s confidence 439 (88) 402 (93) 0.033
Item 4: Involved you in entire appointment 438 (88) 408 (94) 0.002
Item 5: Examination of pet 436 (88) 400 (92) 0.031
Item 6: Explained treatment and procedures 432 (87) 408 (94) < 0.001
Item 7: You understood the costs 398 (80) 362 (83) 0.212
Item 8: Involved you in decisions 436 (88) 399 (92) 0.043
Item 9: Discussed options with you 422 (85) 398 (92) 0.002
Item 10: Discussed cost with you 373 (75) 344 (79) 0.141
Item 11: Interested in your opinion 420 (85) 404 (93) < 0.001
Item 12: Information you received 429 (87) 397 (91) 0.016
Item 13: Addressed your concerns 438 (88) 406 (94) 0.006
Item 14: Recognized role of pet in your life 424 (86) 398 (92) 0.003
Item 15: Amount of time spent with you 428 (86) 402 (93) 0.002

P values were calculated with a χ2 test.

Intervention effect

The final mixed logistic regression model to assess the effect of the intervention on client satisfaction included 865 participants with complete responses to the predictor variables. The odds of clients being completely satisfied were significantly higher during the postintervention period than during the preintervention period (OR = 1.56; 95% CI, 1.17 to 2.06; F = 9.42; P < .001), after adjusting for visit type (F = 2.95; P = .032) and duration of the veterinarian-client relationship (F = 6.99; P = .008).

Predictor variables

The odds of being completely satisfied were significantly higher for recheck appointments than for preventative care appointments (OR = 1.71; 95% CI, 1.09 to 2.68; P = .02) and health problem appointments (OR = 1.99; 95% CI, 1.26 to 3.14; P = .003) but did not differ significantly from the odds for emergency appointments (OR = 1.97; 95% CI, 0.83 to 4.70; P = .12). The odds of being completely satisfied with a preventative care appointment were not significantly different from the odds of being completely satisfied with a health problem appointment (OR = 1.16; 95% CI, 0.85 to 1.58; P = .34) or an emergency appointment (OR = 1.15; 95% CI, 0.52 to 2.56; P = .73). The probability of being completely satisfied increased by 0.52 (95% CI, 0.50 to 0.53; P = .008) with each 1-year increase in the duration of the veterinarian-client relationship.

Discussion

The present study used a multipractice, preintervention-postintervention design to assess the impact of in-practice communication skills training on client visit satisfaction, and significantly more clients were completely satisfied with their visit following the intervention. These findings suggested that client satisfaction can be positively impacted by formal in-practice clinical communication training. Several factors might contribute to the success of the intervention, including practice culture and leadership, a client-centered approach, experiential education, and effective use of communication skills.

The veterinary practice group in the present study prided itself on demonstrating high regard for relationships between the veterinary team and clients and fostering a culture of compassionate communication. These preexisting values aligned closely with the client-centered approach taught in the curriculum. The hospital leadership team, including practice owners, medical directors, and hospital managers, was invested in putting these beliefs into action. The entire veterinary team participated in the intervention, promoting a shared vision, goals, and knowledge, which was proven to improve outcomes.18 Veterinary team members conversed with a common communication skill language during and between learning sessions. All of this established a consistent, unified, and team-based approach enhancing receptivity and adoption of communication skills during the intervention.

The intervention was based on the principles of client-centered care. Analogous to patient-centered care in human medicine,19 client-centered care promotes a collaborative, interactive process between the client and veterinarian to encourage optimal clinical outcomes for the animal.20 A client-centered approach strives to achieve a balance of power and promote dialogue between the veterinarian and client, moving away from the traditional veterinarian-centered model.2123 This concept was taught to the veterinary teams in the present study through skills-based training, encouraging team members to explore beyond the biomedical aspects of patient care to include dialogue related to the client and environment. This method entails building strong relationships with clients and developing an understanding of the client’s perspective and the greater environment in which the client and pet live. A prominent cause of breakdowns in veterinarian-client communication that impacts the client experience is a client feeling that their concerns have not been heard.6 Clients want their veterinarian to take time to listen to what they have to say, including the concerns and observations that clients have about their pet.68 A client-centered approach is more likely to meet client expectations and enhance client appointment-specific satisfaction.

Best practices for teaching clinical communication were employed in the present study.24 Evidence suggests that experiential teaching techniques and a learner-centered approach are more likely to produce communication-specific behavioral changes than didactic approaches.24 The training incorporated a skills-based communication curriculum focusing on a subset of the CCG communication skills and employed active learning techniques, including small group discussion, skills drills, and role-play as well as individual observation, feedback, reflection, and coaching. Continuity was maintained between sessions by conducting communication rounds every other week throughout the 15-month intervention period.

It is likely that the interactive teaching methods used in the present study advanced communication behaviors among participants,25,26 a hypothesis that will be examined as part of the larger study. The significant increase in many of the individual CSQ items could be attributed to the communication skills taught throughout the intervention (Appendix 2). For example, significantly more clients were completely satisfied with their veterinarian’s explanation of treatments and procedures after the intervention. The communication training incorporated 3 communication skills specific to explanation and planning that aided veterinarians with their process of providing information to clients. Assessing a client’s knowledge provides a veterinarian with an understanding of what level of information to provide and promotes an explanation tailored to the client’s current comprehension.16 Using easily understood language involves using terminology that is appropriate and relevant to the client.16 Veterinarians were taught to use lay language and define terminology in a way clients could comprehend. Finally, chunk and check entails sharing small pieces, or chunks, of information to allow the client to absorb the material conveyed to them more easily.16 This is followed by an open-ended question, or check, to ask what needs to be clarified or what further information is needed. Clients want their veterinarian to provide information that is tailored to the client and to communicate through the use of easy-to-understand language.6,8,9 It is anticipated that these skills enhanced clients’ satisfaction with veterinarians’ communication of treatments and procedures and should be incorporated into fundamental communication skills training in practice.

Two of the 3 CSQ items that did not demonstrate a significant change in the proportion of clients that were completely satisfied before versus after the intervention were related to cost conversations. Clients were asked to rate “How well you understood the cost today” and “The veterinarian’s discussion of cost with you” from 0 to 100. Financial conversations are recognized as a communication challenge for veterinarians.6,27 Content-specific training on financial discussions was not provided in the communication skills training in the present study. Therefore, it is possible that veterinarian participants did not recognize the opportunity to apply the communication skills taught to financial discussions. Future communication skills education could incorporate communication content (what is said) in addition to process skills (how it is said) to address ongoing deficiencies related to cost discussions. Comparable to the findings of Coe et al,15 these cost-of-care questions were also the items with the greatest number of missing values owing to either a lack of response (nonresponse) or inability to respond (unable to assess). Further research into the role that cost discussions play in client satisfaction is recommended, in that the 2 cost-related items in the CSQ had the lowest levels of satisfaction among clients in the present study, consistent with findings of previous studies13,15,28 that used the CSQ.

In the present study, the odds of clients being completely satisfied were significantly higher for recheck visits than for preventative care and health problem appointments, which is a communication appointment context not previously studied. One study29 identified a strong positive correlation between veterinarians’ provision of information and tangible support regarding a health problem and client satisfaction. Recheck visits often provide veterinarians the opportunity to provide health problem support to clients, which may explain the finding of increased satisfaction with recheck appointments in the present study. Additionally, Coe30 found that client satisfaction was greater for clients who visited their regular veterinarian (ie, responded “yes” to the question “I use this veterinarian regularly”), compared with clients having a first-time visit. It could be hypothesized that attending a recheck appointment further built rapport and contributed to the veterinarian-client relationship, leading to increased odds of clients being completely satisfied with recheck visits in the present study. Clients in the present study identified 15% of appointments as a recheck appointment. Further investigation of the content and tone of recheck appointments would be beneficial to foster a greater understanding of factors that contribute to client satisfaction with recheck appointments.

Congruent with findings of a previous case-based study,13 there was no significant difference in proportion of completely satisfied clients between preventative care and health problem visits. Previously, Shaw31 identified disparities in the content of medical dialogue and emotional tone of appointments between preventative care and health problem appointments. Preventative care visits exhibited more social talk with the client as well as verbal interaction with the pet,31 both of which were factors identified by Coe30 to be positively associated with client satisfaction. Although it could be expected that client satisfaction would differ between preventative care and health problem appointments, results of the current and previous studies do not corroborate this hypothesis. Enhanced awareness of how client expectations differ on the basis of appointment type would provide valuable insight into how veterinarians could use communication skills more effectively in various contexts.

Findings of the present study suggest that client satisfaction increases as the veterinarian-client relationship develops over time. Similarly, client satisfaction has been found to be greater for clients who visited their regular veterinarian, compared with clients completing a first-time visit.30 It is possible that clients who are satisfied with their first visit are more likely to maintain an ongoing relationship with that veterinarian, compared with clients who were unsatisfied with their first visit, causing the potential for selection bias. Research conducted in human medicine identified patient trust and good interpersonal relationships as predictors of patient satisfaction and loyalty to their primary care physician.32 Maintaining long-term veterinarian-client relationships is not only beneficial for clients, but is positively associated with veterinarians’ satisfaction with the veterinarian-client-patient relationship,33 with parallel results exhibited in human medicine.34,35 Clinical communication training provides veterinarians with the skills to support, build, and strengthen the veterinarian-client-patient relationship and increase client satisfaction, which may be particularly valuable during first-time visits, when familiarity with the veterinarian is at its lowest.

There are several limitations to consider in relation to the findings of the present study. The use of client-reported appointment type introduced the potential for misclassification bias. However, misidentification of appointment type was minimized by the survey administrators’ clarification. The visual analog modification to the CSQ scale resulted in negatively skewed data, with a high proportion of respondents selecting the maximum score.36 The original unbalanced 6-point adjectival response scale provided distinguishing choices for 4 positive and 2 negative responses (ie, poor, fair, good, very good, excellent, could not be better) and achieved greater variability and construct validity than the present study.15 Thus, it is preferable to use the original scale’s unbalanced 6-point adjectival response format to allow clients to distinguish between positive responses.

Measurements on a visual analog scale have been found to be sensitive to small differences.37 This may explain the finding that overall CSQ scores with no missing values differed significantly from overall CSQ scores with at least 1 missing value. It was not possible to further explore participant-related factors associated with missing values, because the survey output did not differentiate between missing values and respondents’ selection of “unable to assess” on the modified-CSQ items. Only complete CSQs were included in final analyses, which reduced the total sample size considerably (930/1,227 [76%] surveys). The final data set did not meet the criteria of 60 CSQs/veterinarian, as calculated by Coe et al15 to differentiate client satisfaction between veterinarians (preintervention, 49 to 62 CSQs/veterinarian; postintervention, 39 to 55 CSQs/veterinarian). Although there was sufficient power to assess the impact of the intervention on client visit satisfaction, with a sample size of 9 veterinarians from 4 practices owned by a single practice group in 1 geographic region, the results of the present study may not be generalizable to a wider population of veterinarians.

Owing to employee turnover, it was not possible to follow veterinary team members throughout the intervention. It took several on-site learning sessions for the veterinary team to adopt the communication skills. During the third visit, the primary investigator (JRS) repeated the sessions on initiation, agenda setting, and history gathering (Appendix 1) because behavioral changes were not observed during individual coaching. This repeated session resulted in the leadership team clearly setting expectations for active engagement in the intervention and anecdotal application of the communication skills.

In the present study, 9 veterinarians and their veterinary teams across 4 companion animal practices owned by a single practice group in Austin, Texas, underwent a 15-month, in-practice skills-based communication training program that focused on promoting client-centered communication techniques. Findings indicated that skills-based communication training increases appointment-specific client satisfaction with a veterinarian in that the odds of a client being completely satisfied with their visit were significantly greater after than before the intervention. The positive association found between client satisfaction and duration of the veterinarian-client relationship highlights the importance of developing strong veterinarian communication skills to build long-lasting relationships with clients.

Acknowledgments

Funding for this study was obtained through a gift from Zoetis to the Colorado State University College of Veterinary Medicine and Biomedical Sciences. Zoetis did not have any involvement in study design, data analysis and interpretation, or writing of the manuscript.

The authors declare that there were no conflicts of interest.

References

  • 1.

    Adams CL, Conlon PD, Long KC. Professional and veterinary competencies: addressing human relations and the human-animal bond in veterinary medicine. J Vet Med Educ 2004;31(1):6671. doi:10.3138/jvme.31.1.66

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 2.

    Bristol DG. Using alumni research to assess a veterinary curriculum and alumni employment and reward patterns. J Vet Med Educ 2002;29(1):2027. doi:10.3138/jvme.29.1.20

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3.

    Brockman BK, Taylor VA, Brockman CM. The price of unconditional love: consumer decision making for high-dollar veterinary care. J Bus Res 2008;61(5):397405. doi:10.1016/j.jbusres.2006.09.033

    • Search Google Scholar
    • Export Citation
  • 4.

    Brown JP, Silverman JD. The current and future market for veterinarians and veterinary medical services in the United States. J Am Vet Med Assoc 1999;215(2):161183.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 5.

    Lue TW, Pantenburg DP, Crawford PM. Impact of the owner-pet and client-veterinarian bond on the care that pets receive. J Am Vet Med Assoc 2008;232(4):531540. doi:10.2460/javma.232.4.531

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 6.

    Coe JB, Adams CL, Bonnett BN. A focus group study of veterinarians’ and pet owners’ perceptions of veterinarian-client communication in companion animal practice. J Am Vet Med Assoc 2008;233(7):10721080. doi:10.2460/javma.233.7.1072

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 7.

    Stoewen DL, Coe JB, MacMartin C, Stone EAE, Dewey C. Qualitative study of the communication expectations of clients accessing oncology care at a tertiary referral center for dogs with life-limiting cancer. J Am Vet Med Assoc 2014;245(7):785795. doi:10.2460/javma.245.7.785

    • Search Google Scholar
    • Export Citation
  • 8.

    Janke N, Coe JB, Bernardo TM, Dewey CE, Stone EA. Pet owners’ and veterinarians’ perceptions of information exchange and clinical decision-making in companion animal practice. PLoS One 2021;16(2):e0245632. doi:10.1371/journal.pone.0245632

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 9.

    Stoewen DL, Coe JB, MacMartin C, Stone EA, Dewey CE. Qualitative study of the information expectations of clients accessing oncology care at a tertiary referral center for dogs with life-limiting cancer. J Am Vet Med Assoc 2014;245(7):773783. doi:10.2460/javma.245.7.773

    • Search Google Scholar
    • Export Citation
  • 10.

    McArthur ML, Fitzgerald JR. Companion animal veterinarians’ use of clinical communication skills. Aust Vet J 2013;91(9):374380. doi:10.1111/avj.12083

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 11.

    Kanji N, Coe JB, Adams CL, Shaw JR. Effect of veterinarian-client-patient interactions on client adherence to dentistry and surgery recommendations in companion-animal practice. J Am Vet Med Assoc 2012;240(4):427436. doi:10.2460/javma.240.4.427

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 12.

    Cornell KK, Coe JB, Shaw DH, Felsted KE, Bonvicini KA. Investigation of the effects of a practice-level communication training program on veterinary health-care team members’ communication confidence, client satisfaction, and practice financial metrics. J Am Vet Med Assoc 2019;255(12):13771388. doi:10.2460/javma.255.12.1377

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 13.

    Shaw JR, Barley GE, Broadfoot K, et al. Outcomes assessment of on-site communication skills education in a companion animal practice. 2016;249:41932. doi:10.2460/javma.249.4.419

    • Search Google Scholar
    • Export Citation
  • 14.

    McArthur M, Fitzgerald J. Evaluation of a communication skills training program for companion-animal veterinarians: a pilot study using RIAS coding. J Vet Med Educ 2016;43(2):111125. doi:10.3138/jvme.0215-016R2

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 15.

    Coe JB, Adams CL, Eva K, Desmarais S, Bonnett BN. Development and validation of an instrument for measuring appointment-specific client satisfaction in companion-animal practice. Prev Vet Med 2010;93(2-3):201210. doi:10.1016/j.prevetmed.2009.10.005

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 16.

    Adams CL, Kurtz S. Skills for communicating in veterinary medicine. Otmoor Publishing; 2016.

  • 17.

    Kraut AS, Omron R, Caretta-Weyer H, et al. The flipped classroom: a critical appraisal. West J Emerg Med 2019;20(3):527536. doi:10.5811/westjem.2019.2.40979

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 18.

    Bolton R, Logan C, Gittell JH. Revisiting relational coordination: a systematic review. J Appl Behav Sci 2021;57(3):290322. doi:10.1177/0021886321991597

    • Search Google Scholar
    • Export Citation
  • 19.

    Speck P, Higginson I, Addington-Hall J. Spiritual needs in health care. BMJ 2004;329(7458):123124. doi:10.1136/bmj.329.7458.123

  • 20.

    Shaw JR, Adams CL, Bonnett BN. What can veterinarians learn from studies of physician-patient communication about veterinarian-client-patient communication? J Am Vet Med Assoc 2004;224(5):676684. doi:10.2460/javma.2004.224.676

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 21.

    Roter D. The enduring and evolving nature of the patient-physician relationship. Patient Educ Couns 2000;39(1):515. doi:10.1016/S0738-3991(99)00086-5

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 22.

    Shaw JR, Bonnett BN, Adams CL, Roter DL. Veterinarian-client-patient communication patterns used during clinical appointments in companion animal practice. J Am Vet Med Assoc 2006;228(5):714721. doi:10.2460/javma.228.5.714

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 23.

    Cornell KK, Kopcha M. Client-veterinarian communication: skills for client centered dialogue and shared decision making. Vet Clin North Am Small Anim Pract 2007;37(1):3747. doi:10.1016/j.cvsm.2006.10.005

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 24.

    Kurtz SM, Silverman J, Draper J. Teaching and learning communication skills in medicine. 2nd ed. Radcliffe Medical Press; 2005.

  • 25.

    Smith S, Hanson JL, Tewksbury LR, et al. Teaching patient communication skills to medical students: a review of randomized controlled trials. Eval Health Prof 2007;30(1):321. doi:10.1177/0163278706297333

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 26.

    Lane C, Rollnick S. The use of simulated patients and role-play in communication skills training: a review of the literature to August 2005. Patient Educ Couns 2007;67(1-2):1320. doi:10.1016/j.pec.2007.02.011

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 27.

    Coe JB, Adams CL, Bonnett BN. A focus group study of veterinarians’ and pet owners’ perceptions of the monetary aspects of veterinary care. J Am Vet Med Assoc 2007;231(10):15101518. doi:10.2460/javma.231.10.1510

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 28.

    Corah L, Mossop L, Dean R, Cobb K. Measuring satisfaction in the small animal consultation and its relationship to consult length. Vet Rec 2020;187(11):446. doi:10.1136/vr.105910

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 29.

    Kedrowicz AA. Clients and veterinarians as partners in problem solving during cancer management: implications for veterinary education. J Vet Med Educ 2015;42(4):373381. doi:10.3138/jvme.0315-048R

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 30.

    Coe J. Communication During Veterinarian-Client-Patient Interactions in Companion Animal Practice. Thesis. University of Guelph; 2008.

  • 31.

    Shaw JR, Adams CL, Bonnett BN, Larson S, Roter DL. Veterinarian-client-patient communication during wellness appointments versus appointments related to a health problem in companion animal practice. J Am Vet Med Assoc 2008;233(10):15761586. doi:10.2460/javma.233.10.1576

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 32.

    Platonova EA, Kennedy KN, Shewchuk RM. Understanding patient satisfaction, trust, and loyalty to primary care physicians. Med Care Res Rev 2008;65(6):696712. doi:10.1177/1077558708322863

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 33.

    Shaw JR, Adams CL, Bonnett BN, Larson S, Roter DL. Veterinarian satisfaction with companion animal visits. J Am Vet Med Assoc 2012;240(7):832841. doi:10.2460/javma.240.7.832

    • PubMed
    • Search Google Scholar
    • Export Citation
  • 34.

    Donahue KE, Ashkin E, Pathman DE. Length of patient-physician relationship and patients’ satisfaction and preventive service use in the rural south: a cross-sectional telephone study. BMC Fam Pract 2005;6(1):40. doi:10.1186/1471-2296-6-40

    • Search Google Scholar
    • Export Citation
  • 35.

    Suchman AL, Roter D, Green M, Lipkin M Jr. Physician satisfaction with primary care office visits. Collaborative Study Group of the American Academy on Physician and Patient. Med Care 1993;31(12):10831092.

    • Search Google Scholar
    • Export Citation
  • 36.

    Hessling RM, Traxel NM, Schmidt TJ. Ceiling effect. In: Lewis-Beck MS, Bryman A, Liao TF, eds. The Sage Encyclopedia of Social Science Research Methods Sage Publications; 2004:107.

    • Search Google Scholar
    • Export Citation
  • 37.

    du Toit R, Pritchard N, Heffernan S, Simpson T, Fonn D. A comparison of three different scales for rating contact lens handling. Optom Vis Sci 2002;79(5):313320. doi:10.1097/00006324-200205000-00011

    • PubMed
    • Search Google Scholar
    • Export Citation

Appendix 1

Curriculum for an in-practice communication skills intervention taught to veterinary teams in 4 veterinary hospitals during 7 visits across a 15-month intervention period (April 2018 to June 2019) in a study designed to assess the impact of the intervention on client visit satisfaction.

Visit No. Calgary-Cambridge Guide task taught Communication skills
1 (April 2018) Communication styles
2 (June 2018) Initiating the session, agenda setting Introduction
Agenda setting
Open-ended questions, closed-ended questions
Summary
3 (August 2018) History gathering Reflective listening
Pause
Eliciting client perspective
4 (November 2018) Agenda setting, history gathering (repeated session) Agenda setting
Open-ended questions, closed-ended questions
Summary
Reflective listening
Pause
Eliciting client perspective
5 (January 2019) Explanation and planning Chunk and check
Using easily understood language
Assessing client’s knowledge
Relating to client perspective
6 (April 2019) Relationship building Empathy
Asking permission
Partnership
7 (June 2019) Providing structure, closing the session Logical sequence
Signposts
End summary
Contracts for next steps
Final check

Appendix 2

Communication skills taught in the communication skills intervention hypothesized to impact items on the Client Satisfaction Questionnaire (CSQ).

CSQ Item Communication skills
Item 2: Veterinarian understood reason for visit Agenda setting
Item 4: Involved you in entire appointment Agenda setting
Eliciting client perspective
Relating to client perspective
Assessing client’s knowledge
Chunk and check
Asking permission
Partnership
Item 6: Explained treatment and procedures Chunk and check
Using easily understood language
Assessing client’s knowledge
Item 8: Involved you in decisions Eliciting client perspective
Relating to client perspective
Assessing client’s knowledge
Chunk and check
Asking permission
Partnership
Item 9: Discussed options with you Chunk and check
Eliciting client perspective
Relating to client perspective
Item 11: Interested in your opinion Eliciting client perspective
Partnership
Item 12: Information you received Chunk and check
Using easily understood language
Relating to client perspective
Item 13: Addressed your concerns Agenda setting
Final check
Empathy
Item 14: Recognized role of pet in your life Eliciting client perspective
Relating to client perspective
Empathy
Reflective listening
All Time Past Year Past 30 Days
Abstract Views 762 0 0
Full Text Views 1916 1538 87
PDF Downloads 573 238 22
Advertisement