Showing Uncertainty of Official Statistics

Keywords:uncertainty, visualisation

1.Introduction

Statistics is the science of quantitative estimation of social, economic or natural phenomena. Estimation entails that the resulting figures are not precise: they have measure of uncertainty. In the sciences measurements without a confidence interval indicating their uncertainty are usually not accepted. Strangely enough most Official Statistics are communicated without a measure of uncertainty, even when it is available. For example economic growth and inflation are often presented as being exact, thoughthey form the basis for important decisions. [1] argues that omitting uncertainty in the presentation of official statistics leads their users to think that the data is error-free or lets them conjecture wrong error sizes.Besides the fact that uncertainty may be very difficult to estimate due to systematic or measurement errors, one reason for leaving out uncertainty in the communication is the assumption that users do not appreciate error margins or cannot interpret them. We conducted two user experiments to test these assumptions: an interactive software prototype in which users could select and view open data from a NSI and a carefully designed user survey with synthetic data in which the effect of showing uncertainty on user statements was tested.

1.1.Context

Both user experiments were conducted in the StatMine project. This research project tests several improvements on the Open Data dissemination database of a NSI by implementing an enhancement in a software prototype and testing it on statistical analysts, policy makers and journalists. One proposed improvement is displaying uncertainty intervals with published statistical figures. Furthermore, a more general enhancement was to publish and present interval estimates. Interval estimatesexpand the range of statistical methods that can be used for producing official statistics. For example in statistical disclosure control interval estimates allow more cells to be published than point estimate methods.

We use a simple but general notion of uncertainty: each statistic yhas at least an interval estimate, meaning that has a value that lies with high probability (e.g. 95%) in the interval . In addition a statistic y may have a point estimate indicating it’s most likely value. No further assumptions are made on the shape of the probability density function or the source of uncertainty. It can be a traditional 95% confidence interval using a normality assumption, it may be Bayesian credible interval or it might include systematic and measurement error estimates.

2.Methods

2.1.Experiment 1: Interactive software prototype

The interactive prototype of the StatMine project was adapted to include and display uncertainty intervals on several data sets. A cognitive psychologist conducted a user test with 10 users in which a user answered several questions with the prototype. Each test was preceded and concluded with a detailed interview.

2.2.Effect of displaying uncertainty on user interpretation

A survey (N=108) was conducted which tested for the effect of displaying uncertainty in line and bar charts [2] on user interpretation. A line chart is mostly used for time series and helps a user to detect if the series is decreasing, constant or increasing. A bar chart is mostly used to compare absolute values of different subpopulations: Is the average wage of men higher than the wage of women?Several types of line and bar charts were tested on different datasets.

Each dataset was tested with a different chart type on a different group respondents. Each respondent saw all datasets with all charts types, but each group saw a different combination. For each line chart a respondent had to answer several questions, including a question if the line was increasing (5 certainty levels). For each bar chart a respondent answered multiple questions including if bar A was higher than bar B (5 certainty levels). The last task was, depending on the dataset and the display of error bars, sometimes ambiguous.

The survey included questions on user preference, which were confronted with their performance.

3.Results

3.1.Interactive prototype

The tested users do not react confused, but find the uncertainty margins useful. For example policy makers frame the interval as the range between a worst and a best case scenario, with being the most likely scenario. When uncertainty margins are displayed many users ask what the exact meaning of is, which indicates genuine user interest in those figures. When offered the choice to omituncertainty margins, users prefer to keep them.

3.2.Survey on effect

When uncertainty is displayed in line charts, users are more often correct in detecting a statistically significant trend in the data. Users are overconfident in detecting a trend in a line chart without uncertainty. The Ribbon +line scores best and is the preferred option for displaying time series. However when no point estimates is available, error bar should be chosen.

Table 1. Correctness of trend assessment in line charts

Type / Answers / Trend correct
Ribbon + line / 90 / 88.9%
Error bar / 83 / 85.5%
Error bar + line / 82 / 81.7%
Ribbon / 86 / 80.2%
Line / 91 / 74.%

In bar charts many respondents prefer error bars, although they perform slightly worse than other options (see also [3]): this may be an effect of education.

4.Conclusions

Against common opinion, the statistical analysts, policy makers and journalists appreciate error margins when they are displayed.Displaying uncertainty improves the validity of user assessment of trends in line charts. This opens up the publication of uncertainty measures and interval estimates.

References

[1]Manski, C. F. (2014). Communicating uncertainty in official economic statistics. Technical report, NationalBureau of Economic Research.

[2]Playfair, W. (1786). The Commercial and Political Atlas: Representing, by Means of Stained Copper-PlateCharts, the Progress of the Commerce, Revenues, Expenditure and Debts of England during the Whole ofthe Eighteenth Century.

[3]Sanyal, J., Zhang, S., Bhattacharya, G., Amburn, P., andMoorhead, R. J. (2009). A user study to comparefour uncertainty visualization methods for 1D and 2Ddatasets. IEEE transactions on visualization and computer graphics, 15(6):1209–18.

1