Print version

Research

Major past projects completed by researchers at the Centre for Cognitive Work and Safety Analysis include:

Training Needs Analysis

Many organisations have worked under the assumption that simply purchasing expensive, high-fidelity training devices will reduce training costs, increase levels of skill in the workforce, and reduce the risk of accidents on the job. For two main reasons, these expectations have not always been realised. First, limited attention has been paid to the analysis of training needs in these organisations. Second, even when a training needs analysis has been conducted, little consideration has been given to the systematic specification of training-system requirements that are necessary for satisfying the organisation's training needs. On this project we developed a framework for using Work Domain Analysis (the first phase of Cognitive Work Analysis) to identify training needs and training-system requirements for complex systems. This framework was used to support the acquisition of a training system for Australia's F/A-18 aircraft. Specifically, Work Domain Analysis was used to identify the purposes, values and priorities, functions, and physical resources of the F/A-18 system. Subsequently, these properties of the F/A-18 work domain were used to derive different categories of training needs for F/A-18 pilots and different categories of training-system requirements to meet those training needs. The training-system requirements derived from the F/A-18 Work Domain Analysis were incorporated into the Request for Tender documentation for the acquisition of the F/A-18 training system.

Naikar, N., & Sanderson, P. M. (1999). Work domain analysis for training-system definition and acquisition. The International Journal of Aviation Psychology, 9(3), 271-290. (attachment available on publications page).

Evaluation of Design Proposals

In the early stages of system acquisition, the buyer of a new system typically evaluates several proposals that are submitted by competing manufacturers for the design and development of the intended system. This is an important stage of acquisition because following the selection of one of the proposals, the buyer becomes locked into a particular design concept and further development is focused on modifying or refining the chosen design. On this project, we developed a framework for using Work Domain Analysis (the first phase of Cognitive Work Analysis) to evaluate alternative design proposals for complex systems. This framework was used to evaluate three competing design proposals for Australia's new Airborne Early Warning and Control (AEW&C) system. Specifically, Work Domain Analysis was used to identify the purposes, values and priorities, functions, and physical resources of the new AEW&C system. Subsequently, this framework was used to evaluate the three design proposals in terms of how well they supported the higher-level functions, values and priorities, and purposes of AEW&C. This approach focuses evaluation on military utility rather than solely on technical properties and complements the systems engineering techniques that are typically used for tender evaluation or source selection.

Naikar, N., & Sanderson, P. M. (2001). Evaluating design proposals for complex systems with work domain analysis. Human Factors, 43(4), 529-542. (attachment available on publications page).

Team Design

Existing techniques for team design cannot readily be applied to future systems. On this project, we developed a framework for using Work Domain Analysis and Control Task Analysis (the first and second phases of Cognitive Work Analysis) to design teams for complex future systems. This framework was used to develop a crew concept for Australia's new Airborne Early Warning and Control (AEW&C) system. Specifically, Control Task Analysis was used to analyse the effects of different crew concepts on the activity of team members in a variety of scenarios. Subsequently, Work Domain Analysis was used to evaluate the alternative crew concepts in terms of how well the different patterns of activity supported the purposes, values and priorities, and functions of AEW&C, as well as how effectively the physical resources of AEW&C were shared. This framework resulted in team design that was judged by domain experts as better than the designs they had previously considered for AEW&C. Furthermore, by specifying a team design for AEW&C during the early stages of system development, this framework has allowed us to monitor whether or not the engineering of the technical system will support the proposed crew concept.

Naikar, N., Pearce, B., Drumm, D., & Sanderson, P. M. (2003). Designing teams for first-of-a-kind, complex systems using the initial phases of cognitive work analysis: Case study. Human Factors, 45(2), 202-217. (attachment available on publications page.)

Accident Analysis

Techniques that have previously been used for analysing accidents have produced hundreds of safety recommendations but, with limited time and resources, it is difficult to know where efforts for improvement should begin. AcciMap Analysis is useful for this purpose since, by capturing the interrelationships between causes originating from different parts of the systems, it enables analysts to pinpoint the most influential contributors to the accidents and, therefore, the areas in which efforts to improve safety would be of most benefit. In this project, we used AcciMap Analysis to identify the multiple causes of a number of accidents that have recently occurred in complex, military systems. On the basis of the results we obtained, the Directorate of Flying Safety in the Australian Department of Defence has become interested in adopting AcciMap Analysis as a key framework for accident analysis.

Publications relating to this project are classified.

Training for Error Management

Robust error management within the cockpit is crucial to aviation safety. Crew Resource Management (CRM) focuses on non-technical skills for error management, but the training of technical skills for error detection and error recovery is also a potentially valuable strategy. On this project, we developed a framework for using Control Task Analysis (the second phase of Cognitive Work Analysis) and the Critical Decision Method to identify training strategies for managing human error. This framework was used to identify error-management training strategies for F-111 pilots and navigators. Specifically, Control Task Analysis was used to model the problem-solving difficulties that F-111 aircrew have experienced at critical points in a variety of accidents and incidents. The Critical Decision Method was used to interview F-111 aircrew about their experiences in various incidents in order to develop the models of aircrew problem-solving. These techniques led to an understanding of the problem-solving strategies that F-111 aircrew should practice during training in order to improve error detection and error recovery. In a survey involving several F-111 training instructors, pilots, navigators, and simulator operators, close to 100% of respondents indicated that the training strategies we had identified would be useful for helping them deal with errors on real missions.

Naikar, N., & Saunders, A. (2003). Crossing the boundaries of safe operation: A technical training approach to error management. Cognition Technology and Work, 5, 171-180. (attachment available on publications page).

Analysis of Activity in Future Systems

Future systems pose special challenges that preclude many standard forms of work analysis for modelling activity in those systems. For instance, some techniques rely on specifying activity in terms of stable sets of tasks or work procedures, however, workers will typically develop novel ways of working as they gain experience with a new system. It is therefore difficult to specify the full set of tasks and work procedures before a system is put into operation. In contrast, Cognitive Work Analysis offers a framework for analysing activity in future systems because it focuses on the constraints that shape workers' behaviour in the first place. On this project we explored the use of Cognitive Work Analysis for analysing activity in a future military system called Airborne Early Warning and Control (AEW&C). This approach involved characterising AEW&C activity in terms of a set of work situations, work functions, decisions, and strategies. These properties can be combined in various ways to form the total response of AEW&C actors. This approach is therefore distinct from many standard approaches to work analysis which decompose activity into a relatively small and finite set of tasks.

Naikar, N., & Pearce, B. (2003). Analysing activity for future systems. Proceedings of the 47th Annual Meeting of the Human Factors and Ergonomics Society (pp. 1928-1932). Santa Monica CA: Human Factors and Ergonomics Society. (attachment available on publications page).

Formalising and Evaluating the AcciMap Approach

The AcciMap Approach is a systems-based technique for analysing the causes of organisational accidents. The technique has been used to analyse accidents in a number of complex systems and to identify areas in which safety interventions would be of benefit. Prior to this research, however, there was no standard AcciMap framework and the approach could only be learned by apprenticeship. This project involved two phases - the formalisation of the technique through the development of a standardised AcciMap format and a set of guidelines for conducting AcciMap Analysis, and an investigation into the validity and reliability of AcciMap results. The research was undertaken as PhD project by Kate Branford from the Department of Sociology at the Australian National University and was supervised by Dr Neelam Naikar from DSTO and Professor Andrew Hopkins from the Australian National University.

Branford, K. (2007). An investigation into the validity and reliability of the AcciMap approach. Unpublished doctoral dissertation, Australian National University, Canberra.

Branford, K., Naikar, N., & Hopkins, A. (2009). Guidelines for AcciMap analysis. In A. Hopkins (Ed.), Learning from high reliability organisations (pp. 193-212). Sydney, Australia: CCH Australia.

Theoretical Concepts and Methodology for Work Domain Analysis

Two of the most comprehensive accounts of Work Domain Analysis are provided by Rasmussen, Pejtersen & Goodstein (1994) and Vicente (1999). Without doubt, these texts are invaluable resources for Work Domain Analysis and Cognitive Work Analysis in general. Because of a number of factors, however, these texts can still leave work domain analysis relatively inaccessible to many researchers and practitioners. In particular, the texts appear to reflect somewhat different theoretical approaches to this form of analysis, and do not offer a comprehensive methodology for performing Work Domain Analysis. This project was concerned with the development of a well-defined theoretical approach and methodology for Work Domain Analysis. We addressed a number of conceptual issues relating to this form of analysis, including differences in the approaches of Rasmussen et al. and Vicente, and we developed guidelines for performing work domain analysis. The conceptual issues and guidelines were illustrated with examples from a familiar system, namely, a home.

Naikar, N., Hopcroft, R., & Moylan, A. (2005). Work domain analysis: Theoretical concepts and methodology. DSTO Technical Report (DSTO-TR-1665). Fishermans Bend, Australia: Air Operations Division. (attachment available on publications page).

Concepts, Guidelines, and Case Study for Control Task Analysis

Although Work Domain Analysis, the first phase of Cognitive Work Analysis, is well recognised, relatively little attention has been given to Control Task Analysis, the second phase of Cognitive Work Analysis. On this project, we consolidated the different approaches to control task analysis presented by Rasmussen, Pejtersen & Goodstein (1994) and Vicente (1999). We also extended the basic concepts for Control Task Analysis, in particular, by observing that activity in some systems is best characterised as a combination of work situations and work functions. In addition, we developed a set of guidelines for performing Control Task Analysis. The concepts and guidelines for Control Task Analysis were illustrated with a case study of a complex military system.

Naikar, N., Moylan, A., & Pearce, B. (2006). Analysing activity in complex systems with cognitive work analysis: Concepts, guidelines and case study for control task analysis. Theoretical Issues in Ergonomics Science, 7(4), 371-394. (attachment available on publications page).

Concepts for Cognitive Work Analysis 

Cognitive Work Analysis is recognised, principally, for Work Domain Analysis. Relatively little research has been directed at the remaining four phases of Cognitive Work Analysis, namely, Control Task Analysis, Strategies Analysis, Social Organisation and Cooperation Analysis, and Worker Competencies Analysis. As a result, the concepts underlying these four phases are less well understood. The aim of this project is to examine the key concepts of the remaining stages of Cognitive Work Analysis. This work will provide a starting point for investigating a number of issues relating to these phases of analysis including whether or not they are useful or unique, their relationships to other approaches for work analysis, and any methodological shortcomings.

Naikar, N. (2006). An examination of the key concepts of the five phases of cognitive work analysis with examples from a familiar system. Proceedings of the 50th Annual Meeting of the Human Factors and Ergonomics Society (pp. 447-451). Santa Monica, CA: Human Factors and Ergonomics Society. (attachment available on publications page).