Evaluating the effects of data visualisation techniques on interpretation and preference of feedback reports

Objective To evaluate the different methods of data visualisation and how it affects preference and data interpretation. Design A cross-sectional survey, assessing interpretation and preference for four methods of data presentation, was distributed to participants. Setting Melbourne, Victoria Participants Members of Prostate Cancer Outcome Registry-Victoria (PCOR-Vic) and senior hospital staff in three metropolitan Victorian hospitals. Interventions Different methods of data visualisation. Mainly, funnel plots, league charts, risk adjusted sequential probability ratio test (RASPRT) charts and dashboard. Main Outcome Measure Interpretation scores assessed capacity by participants to identify outliers and poor performers. Preference was based on a 9-point Likert-scale (0 – 9). Results In total, 113 participants responded to the online survey (16/58 urologists and 97/297 senior hospital staff, response rate 32%). Respondents reported that funnel plots were easier to interpret compared to league charts (mean interpretability score difference of 28% (95% CI: 19.2% - 37.0%, P<0.0001). Predictors of worse interpretability of charts in the adjusted model were being a hospital executive compared to a urologist (coefficient= −2.50, 95% CI = −3.82, - 1.18, P<0.01) and having no statistical training compared to those with statistical training (coefficient = −1.71, 95% CI=-2.85, −0.58, P=0.003). Participants preferred funnel plots and dashboards compared to league charts and RASPRT charts (median score 7/9 vs 5/9), and preferred charts which were traffic-light coloured versus greyscale charts (43/60 (71.6%) vs 17/60 (28.3%)). Conclusion When developing reports for clinicians and hospitals, consideration should be given to preference of end-users and ability of groups to interpret the graphs.


26
Prostate Cancer Outcomes Registry -Victoria (PCOR-Vic), achieved a 2:1 benefit-to-cost ratio 27 with capture of just 11% patients diagnosed in Australia (number of participating hospitals 28 nationwide). Theoretically, with full national coverage, the estimated extrapolated benefit-to-cost 29 ratio of the Prostate Cancer Outcomes Registry could be as high as 5:1. [8] 30 It is unclear whether the complex information presented by CQRs is achieving its optimal impact.

31
Studies have shown that healthcare professionals may not be able to interpret data presented to 32 them in visual formats.[9-11] Therefore, there is merit in understanding the interaction between 33 data presentation and interpretation. Data presentation seeks to find the most efficient and 34 effective way of translating raw data into presentable, interpretable and preferably, visually 35 appealing visual aids that can aid a user in eventual decision making. [12] In the context of 36 healthcare, these graphical methods are used by CQRs in clinical reports to inform healthcare 37 providers on quality of care..

38
Quality Indicator reports developed by CQRs are designed to be easily interpretable and are 39 intended to guide good decision-making and improvement in quality of care. The aim of this study 40 was therefore to evaluate the different methods of data visualisation and how it affects preference 41 and data interpretation in two different groups: urologists and senior hospital staff. In addition, we

51
(1) a league chart, which uses the traditional bar chart to rank the performance of healthcare Plot. (C) Risk-Adjusted Sequential Probability Ratio Test. A chart which uses statistical process 64 control to benchmark clinicians which is implemented in the survey. Figure 1D. Business Analytics 65 Dashboard. Dashboard which allowed a summary of all quality indicators provided in the report.

66
The survey consisted of the following three components: 67 Descriptive statistics were used to assess the characteristics of the study participants. Paired t-

97
test was used to compare the difference in mean percentage of questions answered correctly 98 between the two cross sectional charts (funnel plots and league charts). To score participant 99 preference, a median and inter-quartile range was calculated for each chart and categorised as 100 follows: a median of 0-3 was defined as unfavourable, 4 -6 was considered neutral, and 7-9 was 102 relationship between interpretation score (questions answered correctly out of ten) against three 103 predictors; (1) statistical education, (2) self-rated confidence in understanding basic statistical 104 concepts and (3) the mean score of easiness of interpretation based on three charts (funnel plot, 105 league charts and RASPRT). A chi-squared test was performed to test the questions comparing 106 the preferences of colour coding between groups.

107
Sample size calculation 108 Based on the assumption that the 2-sided level of significance was set at 5% and that the standard 109 deviation of the score was 2, a total sample size of 128 was required for a power of 80% to detect 110 at least a clinically significant difference of 1 score between urologists and senior hospital staff.

111
After the projected non-response rate of 60%, we calculated a final target sample size of 320.

112
Data were analysed using STATA 13. Level of significance was set at 5%.

113
The ethical conduct of this multi-site study was approved by the Alfred Health Human Research and dashboards, with each scoring a median score of 7 out of 9. Participants were less likely to 155 prefer league charts and RASPRT with a median score of 5 out of 9. Urologists rated league 156 charts and RASPRT lower than senior hospital staff (median score of 3 and 3.5 vs 5). 157

187
Our results, which show high interpretation scores for funnel plots, may also be biased by our 188 sampling population. While league charts are used in 80% of registry reports in Australia[18] and accuracy.
[20] The CQR report distributed by PCOR-VIC has 12 funnel plots and one league chart 191 in its performance report. The discrepancy in results may be due to urologists and senior hospital 192 staff being unfamiliar with league charts. A similar reason can be inferred for RASPRT, given that charts lower when compared to funnel plots because they were unfamiliar to respondents. As

195
shown in previous studies, training can be successfully delivered to assist in the interpretation 196 and integration of statistical control charts such as RASPRT and funnel plots into practice. [21][22][23][24] 197 Urologists were more capable of correctly interpreting data presented in CQR reports when 198 compared to senior hospital staff, even after adjusting for statistical knowledge and confidence.

199
This likely reflects the fact that the urologists who participated in the study receive clinician-level

200
CQR reports on a six-monthly basis, while senior hospital staff are less likely to have the same 201 level of exposure to these reports and, by default, to these types of graphs.

202
After adjusting for other predictors, statistical education was found to be significant in its effect on 203 the interpretation score. Effective data visualisation seeks to present highly complex data using

253
Data displayed as funnel plots among urologists and senior hospital staff was superior in 254 identifying outliers in cross-sectional data and was preferred among this cohort when compared 255 to league charts. However, funnel plots display data at one point in time, and future research is 256 required to understand how to best ensure that data displaying progress over time to healthcare 257 professionals is both statistically accurate and easy to interpret. Looking to other industries, such 258 as business and marketing, may provide innovative approaches for displaying complex 259 information which may be applied in health.
260 distributing the survey and to Melissa Gillespie for administrative support.