Technical report | Visualising Uncertainty for Decision Support

Abstract

Uncertainty is inherent in all real world settings and it creates ambiguities that make decision making complex and difficult. For decades, uncertainty visualisation has been a prominent topic in the research of military decision making. Even if data is free from uncertainty, errors can occur in the process of turning the data into the ‘picture’. Ignoring the fact of information uncertainty could lead to severe consequences in the military domain. This report presents an overview of the theoretical concepts and definitions of uncertainty, and the uncertainty visualisation techniques that have been investigated to date. It discusses the literature on the impact of uncertainty visualisation on decision making, and introduces recommended guidelines and systematic strategies for uncertainty visualisation. Finally, it discusses how this work can be applied to support situation awareness and decision making in the Australian Defence Force’s Joint Operations Command. 

Executive Summary

Military commanders are typically required to make critical decisions and develop plans, using large and complex collections of data, in a limited time frame. They must do this without precise knowledge about the operating environment or the intent, capabilities or location of the adversary. It is important that commanders understand the associated uncertainties so that they can understand and mitigate the operational risks involved in this inherently risky enterprise. Therefore it is important for decision support tools to make users aware of the uncertainties in the information being displayed in an appropriate and timely manner, while avoiding information overload. However, most visualisation techniques have been designed around the assumption that the data being visualised is free from uncertainty. This report reviews the concepts, techniques, and effectiveness of visualisation approaches for uncertainty presented in the literature, and looks at how these could be applied to enhance situation awareness in the Joint Operations Command (JOC).

Uncertainty can come in many forms, with 11 different types of uncertainty discussed in the literature:

• Accuracy – the difference between observation and reality
• Precision – the quality of the estimate or measurement
• Completeness – the extent to which information is comprehensive
• Consistency – the extent to which information elements agree
• Lineage – the pathway through which information has been passed
• Currency – the time span from occurrence to information presentation
• Credibility – the reliability of the information source
• Subjectivity – the extent to which the observer influences the observation
• Interrelatedness – the dependence on other information
• Experimental – the width of a random distribution of observations
• Geometric – the region within which a spatial observation lies.

Each of these different types of uncertainty applies to different types of information, and can be quantified, and thus represented, in different ways. These uncertainties can be introduced at any stage during an information processing pipeline:

• Acquisition – introduced by the measurement or sampling processes
• Transformation – introduced by processing algorithms or fusion processes
• Visualisation – introduced by visualisation artefacts or filters.

Indeed, the act of trying to account for, quantify, and visualise uncertainties could potentially introduce more uncertainties into a system, and so it is important to understand the nature of any uncertainties and their impact on the quality of decision making. It is particularly important to ensure that attempting to represent uncertainty does not introduce artefacts that obscure, clutter, or interfere with the information to be displayed. Visualisation approaches that have been used for representing uncertainty fall into two general categories:

• intrinsic representation techniques that integrate uncertainty by varying the appearance of the data (e.g. shape, texture, brightness, opacity and hue)
• extrinsic representation techniques that add geometry to describe the uncertainty (e.g. arrows, error bars and charts).

The choice of visualisation approach depends on the nature of the uncertainty and the application context. For geospatial contexts, five intuitive categories of uncertainty representation have been suggested:

• modification of graphical attributes, such as colour, texture, blurring, and opacity
• addition of artefacts, such as glyphs, contours, and iso-surfaces
• animation of graphical attributes to illustrate the expected variability
• non-visual techniques, such as acoustic and haptic feedback coupled with a visual display
• user interaction, such as information pop-ups on mouse hover over a data feature.

Much of the work covered in the literature has focussed on the technical feasibility and implementation of these approaches, with little analysis of their perceptual or cognitive value, and little systematic evaluation done on the general effectiveness of these approaches. Thus, studies need to be tailored to particular user contexts. There have been empirical studies carried out on the effect of uncertainty on decision making, which found marked differences in performance between experienced and inexperienced users. The decision times for experienced users were not affected by uncertainty, while those of inexperienced users were significantly increased. Interestingly, visualisation of uncertainty has been found to improve decision making performance for relatively easy tasks, but not when dealing with more difficult tasks when, perhaps, other considerations dominate. How user experience maps to subjective task difficulty could be an interesting consideration that makes the selection of participants for empirical studies particularly important when studying the effectiveness of uncertainty representation techniques. Clearly, participants in empirical studies need to be representative of the skills and experience of the target user group.

Several empirical studies have also looked at the effects of different uncertainty visualisation approaches on decision making in different contexts. Dynamic uncertainty representation techniques (e.g. animation) were generally found to be less effective than static techniques (e.g. glyphs) in decision making tasks. Somewhat surprisingly, the addition of textual annotations of confidence to a glyph representing the degree of uncertainty was found to give poorer performance than the glyph alone in a target identification task, perhaps indicating information overload. In other studies, different uncertainty representations were also found to suit different user requirements in the same application, suggesting that some degree of tailoring may be required to meet multiple user roles. This should be treated with caution however, as studies have also shown that user satisfaction with information products does not necessarily coincide with improved situational awareness in general, and user preferences for uncertainty representation techniques do not necessarily coincide with improved decision making performance in particular. System metrics and governance frameworks may be needed to help manage this complexity.

Other approaches to uncertainty visualisation could use the method of representation itself to attribute confidence in the information presented. For example, users have greater confidence in information presented textually than the same information presented by a virtual human avatar. Thus, in this approach information with low uncertainty could be presented to the user as text, while information with high uncertainty could be presented to the user by an avatar. Further studies could also explore how users attribute confidence to information presented using other visualisation modalities.

There are limited guidelines available for the development of applications utilising the various approaches considered when representing multiple types of uncertainty in a decision making context. More study is needed in how to depict multiple forms of uncertainty in the same display, and how adding multiple types of uncertainties to visual displays affects the users’ understanding. General guidelines for visualisation can be applied to help formulate an uncertainty visualisation approach suitable for a particular context, once the relevant components of uncertainty, their relationships to the data, and the desired decision making outcomes are understood. User centred design, iterative approaches to development, and methods for assessing relevant performance metrics, are considered crucial given the sensitivity of decision making outcomes to context, user experience, and user roles.

The situational awareness requirements of the Australian Defence Force’s Joint Operations Command (JOC) range across the spectrum of ADF operations to achieve national strategic objectives. The users in JOC need to access, integrate and visualise a diversity of information across the Defence Enterprise including military text messages, unstructured documents, military databases and open-source content from civilian producers. Furthermore, this needs to be tailored to support a variety of dynamic operational needs, because the users are unlikely to be subject matter experts in each of the data sets being integrated. It is crucial that they understand the uncertainties associated with the information being displayed so that they can mitigate operational risks. In this context, the sources of uncertainty include:

• Incompleteness due to parsing errors from poorly formatted structured content
• Geometric uncertainties associated with geospatial content
• Ambiguities and incorrect entity associations from unstructured content
• Incomplete metadata • Inconsistent or incompatible schema and/or standards associated with structured content • Inconsistent information from multiple sources
• Interrelatedness of content and/or corrections
• Currency of content
• Lineage and/or credibility of content
• Incompleteness where data aggregation would raise the classification above the system’s accreditation
• Incompleteness due to the pagination of search results, map scale and/or viewpoint settings in geospatial displays
• Incompleteness due to object clustering used to reduce visual clutter
• Interference between overlapping visualisation layers and/or overlapping symbology.

Some of the approaches discussed in the literature that could be used to visualise these uncertainties include:

• Combining representation techniques such as opacity, blurring, and degraded icons to visualise compounded uncertainty (e.g. geospatial uncertainty, information currency, and status ambiguity in incident reports or emergency events)
• Highlighting inconsistencies in the information presented through colour coding
• Highlighting the credibility/lineage of content through colour coding
• Providing a visual indicator of missing/incomplete information, with elaboration provided through user interaction (e.g. mouse hover/click)
• Providing multimodal content through audible or haptic feedback while interacting with the visual display
• Using different visualisation modalities to confer different levels of confidence in the information presented (e.g. use of an avatar to confer lower levels of confidence in uncertain information).
• Allowing users to choose appropriate uncertainty visualisation techniques that suit their current task and experience, based on user and role models.

Empirical studies and further conceptual refinement are needed to target the particular requirements of JOC users. Significant barriers to these studies will be the availability of experienced users for empirical evaluation, and the dynamic nature of their visualisation requirements across a broad range of information types and associated uncertainties. The dynamic nature of these requirements means that many of these users may be unfamiliar with a particular task or display, which may make it easier to find participants representative of this target group. It also indicates that visualisation of the associated uncertainties has the potential to provide significant benefits to decision making performance in this context.

Finally, how decision making with uncertainty translates to operational risk is an area that requires further study. In the studies presented in the literature the performance of the decision maker was evaluated using metrics such as response time and the user’s confidence in the outcome. Whether the representation of uncertainty translates to a better appreciation of operational risk, and more effective mitigation strategies, has not been considered in this work. Studies that model and evaluate operational risk, and/or include operational red-teaming, could help address this question.