Research report | Effects of Instant Messaging on Recall During Video-Mediated Briefings

ABSTRACT

This report explores the effect of interruption from instant messaging (IM) on memory of a video-teleconference brief. A 4x2 factorial pilot study was conducted with 32 participants (gender balanced), using four levels of interruption and gender as the independent variables. Two videos were presented to participants, one of a single person speaking and one of a pair in conversation. Memory was tested with five minutes free recall after each video and recognition questionnaires 24 hours later. Analysis revealed that women performed better than men in free recall, that requiring participants to respond to questions degraded their free recall, and that interruption caused a significant downward trend in free recall performance and confidence. Recognition performance results were inconclusive, however. We recommend repeating this study with more participants. Results obtained could help advise Defence on the development of usage policies for collaboration technologies, specifically regarding limiting the types and number of sources of interaction and sensory input where possible.

EXECUTIVE SUMMARY

Effective collaboration is a vital component of the success of any team, especially in a military context where lives may depend on the outcome of any collaborative effort. As part of its research in support of collaboration, the Defence Science and Technology Organisation (DSTO) has run a research activity to develop and support 'smart' meeting rooms called Livespaces. When installed in operational military environments and equipped with domain-specific tools, they are known as Command and Control Developmental BattleLabs (Evdokiou et al., 2004, Vernik et al., 2003).

The Livespaces software (Phillips, 2008) that underpins these spaces provides not only control of devices and information services but also forms a foundation for collaboration software, such as tools for instant messaging (IM), collaborative document editing, and screen sharing. The primary mechanism for group communication between BattleLabs is software-based video teleconferencing (VTC). There are many other technologies available in these environments and it is imperative to investigate how distractions from these technologies affect productivity. The literature on attention, interruption, and dual-task performance strongly suggests that performance will degrade with increases in distraction.

This report investigates how increasing levels of interruption from IM, as a secondary task, may affect memory of short video-mediated briefs, such as a Commander's brief delivered via VTC in a Livespace. It extends upon a Master of Sciences project. We hypothesised that:

• performance in recall of verbally delivered content would decrease with increasing interruption,
• confidence in recall of verbally delivered content would decrease with increasing interruption,
• women would have better recall performance than men, and
• experience with technology would ameliorate the decrease in recall performance and confidence.

A 4x2 factorial design experiment was conducted using four levels of interruption from IM and two levels of gender. Thirty two participants (16 female) were assigned to gender-balanced groups, and each group received a different degree of interruption. Participants watched two 5-minute videos as a primary task, while receiving various degrees of IM as a secondary task, and subsequently performed free recall and recognition tests on the content of the video. They were also asked for confidence ratings on their recollections. One video was of a single person telling stories, and the other video was of a couple discussing travel plans, and the IM content consisted of old (irrelevant) news headlines and questions regarding the video content.

Analyses revealed that interrupting participants with questions significantly degraded their recall, that their confidence in recall significantly diminished over time, and also that women had statistically better free recall than men,. Statistically significant downward trends in free recall performance and confidence with increases in interruption were also found. Recognition data was inconclusive, even after rescoring discarding 'unsure' responses, and neither experience with technologies nor age had significant effect on recall performance or confidence. Although not statistically significant, the increase in false positives with increasing interruption may be of particular interest for future studies: participants receiving questions were worse at spotting false statements regarding the content of the videos.

Even with such a small sample size, these results suggest that attempting to engage in IM while listening to someone speak will mean that a significant portion of what is said may be missed, and thus this practice should be avoided. At the very least it may affect confidence in what is recalled. IM may be used as a secondary task for maintaining awareness via broadcast messages that require no response even if the first task is verbal in nature, but requiring a response diverts too much attention to maintain performance on the primary task, when it is a verbal one.

As a pilot study, the results obtained suggest that further investigation is warranted. If interruption from IM, and other technologies, affects memory and confidence in that memory, it is important that Defence develop usage policies that minimise those effects. These policies may be trialled in Livespaces in particular, at least initially, but their use may extend beyond to other collaboration environments used by Defence. Literature on the effects of animation and colour on attention also may suggest revision of policies on the use of combined tickertape and video displays (e.g. CNN news feeds) in work environments.