Technical report | Evaluating the Effectiveness of Game-Based Training: A Controlled Study with Dismounted Infantry Teams

Abstract

Computer games are increasingly being used by armed forces to supplement traditional methods of military training, despite a lack of empirical evidence on their training effectiveness. This report describes a study conducted by DSTO scientists examining the effectivenesss of a desktop computer game to train small teams of dismounted soldiers in infantry tactics, techniques, and procedures. One infantry section received traditional field-based instruction in section attack procedures, the other took part in game-based training using Virtual Battlespace 2. The performance of both sections was measured before, during, and after training. While the performance of the field-based training section improved significantly from pre-training to post-training, the game-based section showed no significant changes in performance. Overall the findings suggest that game-based training is not effective for training novice teams of infantry personnel in section attack procedures; this contrasts with several previous studies which found game-based training to be effective. The implications for using desktop computer games for individual and team training are discussed and recommendations for future research in this area are outlined.

Executive Summary

Computer games are increasingly being used by armed forces to supplement traditional methods of military training. While the potential benefits of these games are well documented; there is little objective evidence to support their perceived training benefits, especially in the area of collective training for dismounted soldiers.

This report describes an experiment examining the effectiveness of Virtual Battlespace 2 (VBS2) in training section attack tactics, techniques and procedures (TTPs). The study was conducted by Defence Science and Technology Organisation (DSTO) scientists under Task ARM 07/163 Training and Preparedness in response to a request from Training Command - Army to evaluate the efficacy of game-based technologies within the Australian Army, and to provide advice on how to best use these tools for training. The aims of the study were to: (a) compare training outcomes for game-based and field-based training; (b) evaluate the effectiveness of VBS2 for training novice infantry teams in section attack; (c) test a methodology for evaluating game-based training; and (d) provide advice to the military customer regarding the study findings and implications.

Two infantry sections took part in the study and were allocated to either game-based or field-based methods of training section attack procedures. Prior to training, a baseline assessment of each section's ability to conduct section attack in the field was undertaken. Following this, one section received eight hours of game-based training using VBS2; the other section received eight hours of field-based training in section attack procedures. All training was conducted by experienced Section Commanders. After training, both sections were again assessed on their ability to conduct a section attack in the field. All assessments of section attack performance were conducted by military Subject Matter Experts (SMEs).

The key findings from the study were as follows:

• Game-based training for section attack was not effective. The performance of the game-based section did not change significantly after training from their pre-training levels. There was no significant evidence that section attack skills learned during game-based training transferred to the field.

• Field-based training for section attack was effective. The performance of the field-based section improved significantly after training. This evidence suggests that the current method of training section attack to novice infantry soldiers is effective.

These outcomes are in contrast with previous studies with dismounted combatants, which have concluded that game-based training is effective when delivered in addition to or in combination with traditional training methods. However, these studies were unable to quantify the relative effectiveness of game-based training and traditional methods, as we have in the current study.

The most likely reasons for the current findings are: (1) the difference between the physical skills used in the VBS2 virtual environment and those used in the real world, and (2) limitations with the game’s virtual environment such as the restricted field of view (which impacted on the ability of the section to maintain situation awareness) and difficulties in target indication, which resulted from poor audio cues. Possible solutions to these current shortcomings could be investigated by future research (e.g., improving the field of view, and developing some form of automatic target indication). It is possible, however, that such solutions may not be sufficient to improve the effectiveness of VBS2 for training this type of military task; the difference in the physical skills used in the virtual environment and those used in the real world (i.e. field environment) may still be the limiting factor.

The key implications of the findings for Defence are that (1) game-based training with VBS2 would not be considered a cost-effective alternative to the current method of training dismounted soldiers in section attack procedures, and (2) the results are likely to generalise to other collective skills/tasks undertaken by dismounted soldiers, including potentially the preparation of trained soldiers prior to deployment. While the current findings do not support the use of VBS2 for training section attack procedures, it is possible that such computer games might be effective for training other military tasks, such as mission rehearsal and terrain familiarisation. Consequently, it is suggested that alternative applications for such tools be investigated and evaluated as part of future research.

Recommendations for future research are as follows:

 Identify the military tasks that are most suitable for game-based training.

Examine the optimal mix of game-based and traditional training.

Conduct all future studies in this area with a similar level of scientific rigour to that employed in this study.