Aspiring to the gold standard of simulation

Asking the right questions in the right order to get the most out of simulations.

Military simulations are getting bigger and better, and ever more important as Defence leaders incorporate their predictions in making decisions, particularly around force modernisation. But simulations can be expensive and time-consuming to execute, and with higher-fidelity comes increased complexity. Mathematician Dr Andrew Gill, an operations analyst at DSTG, believes more time spent up front considering the purpose of a simulation analysis would be time well spent. He was recently awarded a DSTG Chief Defence Scientist Fellowship to conduct a research program titled “Defence Simulation Analytics: Designing Statistically Efficient Simulation Experiments” and is now on a mission to improve the way analysts interrogate military simulations.

Dr Gill has spent his career at DSTG applying quantitative methods to the study of military operations. As an operations analyst, Dr Gill says one often becomes a ‘jack of all trades’, and over two decades he has developed a toolkit of simulation and data analysis techniques.

 “Analysis of simulations is as critical as the simulations themselves,” he says. “DSTG is a big user of simulations and the impetus for this fellowship was the desire to build up a world-class simulation analytics capability for Defence. A simulation is a quantitative approximation of an aspect of Defence operations. Think of the simulation as an oracle. If you’re not careful then you may have to ask it more questions or you may not even get the right answer. My aim is to come up with the most efficient set and schedule of questions to ask the oracle to get the answer I need.”

Black and white box simulations

According to Dr Gill, most Defence simulations are high fidelity, with a vast array of input factors, but an empirical rule of thumb is that 80% of the results are caused by only 20% of the factors. A screening experimental design and analysis allows a scientist to reveal these critical variables.

Advising Defence that there are only, for example, five factors that really matter when considering a certain aspect of a military operation can in itself often be a valuable result. But the simulation often remains a ‘black box’, with analysts unable to articulate what’s going on inside the box.

A deeper level of simulation analytics involves characterisation. That’s where you make an explicit model of the internals of the black box relating inputs to outputs. The analyst develops equations that can then reveal the contributions of subsystems within the system; they create a model of the simulation, known as a meta-model.

Understanding how the sub-systems interact and contribute to operational effectiveness is valuable in developing advice for military forces, which operate as a system of systems.

For example, in analysing combined arms operations, Defence may want to know how a particular trade-off between subsystems (e.g. armour vs. mobility) affects the results. Proper design and analysis of simulation experiments enable insights to be gained about the marginal contributions of sub-systems to overall operational effectiveness, as well as comparatively assessing competing system alternatives.

The importance of being efficient

According to Dr Gill, designing and interrogating simulations efficiently is sometimes not obvious or intuitive. An inefficient experiment versus an efficient one may be the difference in meeting a deadline to provide advice to the client.

While computing performance continues to increase, Dr Gill argues that the connective nature of simulation analytics means it will remain important to be smart in how we design these experiments.

“An efficient simulation experiment can reveal a broader set of insights. It’s hard to make statistics sexy, but to get the most out of our simulations they are important and impactful.”

Most of his fellowship program will be theory-based, with testing carried out on synthetic data. The rubber will hit the road in the application of the new techniques in support of a major Defence acquisition project.

More simulations, together

Throughout the fellowship, Dr Gill plans to collaborate closely with local, national and international simulation analytics experts. While his focus till now has been on combat simulations for Army, simulations are used by DSTG analysts to study all services and the results of Dr Gill’s fellowship will be applicable to them all. “My mission will be complete when all of our simulation analysis is enabled by the gold standard of statistical efficiency – in essence, producing ‘more juice, with less squeeze’.”