1 Introduction

Naturalistic decision making (NDM) research has developed over the past three decades to become a mainstream applied research paradigm. Originally inspired to take research out of the laboratory and into the real world, the methods and approaches from recognition-primed decision making to macrocognition have become mainstays in the researchers’ and practitioners’ toolkits (Schraagen et al. 2008a; Klein 2010; Hoffman et al. 2014; Chaudet et al. 2015; Ward et al. 2016; Gore et al. 2017).

NDM models, tools and techniques have been applied in domains as diverse as aviation and aerospace, banking, energy production and distribution, defence, ground transportation, nuclear, manufacturing, maritime, medicine, oil and gas, and rail (Gore et al. 2015; Gore and Ward 2017, 2018). NDM methodologies have moved from focussing on individuals to now also working at the level of entire systems, the constraints shaping behaviour and the culture of organisations. The aim of this special issue is to provide researchers and practitioners with an opportunity to present and discuss contemporary, forecasted, and required paradigm shifts for NDM.

2 The NDM conference series

Much of the work included in this special issue was previously presented at the bi-annual NDM conference series, however, for the purposes of our call for papers we also invited researchers from all disciplines and all countries to contribute. This resulted in a greater number of contributions to the special issue than we could accommodate. Where papers were deemed of a high level (but not quite fitting with the NDM theme) they were re-assigned for consideration for the standard issue of cognition technology and work for further consideration.

The conference series commenced in 1989 (Klein et al. 1993; see Table 1 for a summary of the conferences and key publications) and the thirteenth iteration (‘NDM13’) was hosted in 2017 at the University of Bath, with practitioners and academics from across the globe presenting and attending. The main topic of the Bath conference, decision making under uncertainty—was intended to revisit the original focus of our self-organising community of practice (Lave and Wenger 1991), which commenced with a need (and scientific curiosity) for exploring cognition in challenging and complex environments.

Table 1 The history of the International Naturalistic Decision Making Conferences

The papers presented at the conference continued to develop many of the original NDM concepts, as well as considering both familiar (e.g., health, emergency services, intelligence analysis and defence), and newer (e.g., cybersecurity, sensitive policing, and deception). Professional domains of practice. Methodological refinements were also evident as researchers continued to report innovation in their use of cognitive task analysis, visual analytics and technological integration. Attention to sensemaking, situation assessment and further unpacking of metacognition theory and model developments were also highlighted at the conference alongside research in new areas.

With an almost 70 submissions, the final programme featured 25 long papers, 25 posters, three panels, eight invited and keynote speakers, and ten PhD papers (Gore and Ward 2017), the conference covered a wide variety of topics including.

  • Decision making under stress and uncertainty.

  • Methods to study and support rapid decision making.

  • Designing visualisations and user interfaces to improve sense making.

  • NDM in security-related contexts.

  • Designing more effective human–computer planning systems.

  • Bringing evidence-based decision making to bear in civilian and government agencies.

  • Macrocognition and adaptive performance.

  • Complexity in new domains.

The papers in this special issue were selected for inclusion after a competitive review process. Groenewald et al. (2018) lead the special issue with a unique examination of sensemaking in British and Belgian police intelligence analysis. A descriptive, practical, hybrid argumentation model is provided to assist with the formulation of defensible assessments in uncertain sensemaking environments. The intelligence analysts they examined were working with sophisticated visual analytical software in uncertain sense-making environments and were required to create exhibits (as evidence) for a court of law or as an input for intelligence-led policing. The analysts evaluated a low-fidelity prototype resembling the first-order argumentation concepts of an initial argumentation model. The evaluation assessed the applicability and practicality of the concepts within the model. The preliminary results presented indicate that most of the first-order argumentation concepts are both applicable and practical and that the participants would use such concepts to construct their rationale from the onset of an analytical activity, if it were included as part of a software application. It will be interesting to see how this innovative work develops further interested readers should follow the Visual Analytics for sensemaking in Criminal Intelligence Analysis (VALCRI; see http://valcri.org/) project, for future developments.

A further example of innovation in examining new domains is provided by Lefford and Thompson (2018), who examine naturalistic artistic decision-making and metacognition in the music studio. They note that such creative work is characterized by uncertainty, risk, a lack of clearly definable goals, and in the case of music production, a complex socio-technical working environment that brings together a diverse group of specialized collaborators. In music production, there is a professional role explicitly tasked with taking decisions—the (record) producer. The producer, as a creative collaborator, is differentiated as a problem-solver, solution creator and goal setter. This investigation looks at the producer’s metacognitive abilities for reflecting on the nature of problems and decisions. An important challenge for this study was to develop methods for observing decision-making without unrealistically reducing the amount of uncertainty around outcomes or creative intention within a studio production. In the face of that, a method is proposed that combines socio-cultural musicology with cognitive approaches, and uses ethnographic data. Preliminary findings shed light on how the producer self-manages decisions and interactions with, and in response to, the production environment; how decisions and actions sustain collaboration; how experience is utilized to identify scenarios and choose actions; and the kinds of strategies employed and their expected outcomes. Findings provide evidence that skilled performance of production tasks involves a considerable degree of metacognitive reflection.

Gore et al. (2018a), report a unique collaborative, longitudinal, academic practitioner project which aimed to elicit, document, transfer and accelerate the cognitive expertise of engineering professionals working with the manufacture and management of petroleum additives. Engineering experts were trained by the authors of the paper, (academic psychologists), to use applied cognitive task analysis (ACTA) interview techniques to elicit and document the cognition of their expert peers. Results had high face validity for practitioners, who elicited hot/sensory-based cognition, a number of perceptual skills and mental models, highlighting undocumented context-specific expertise. A peer review of findings, combined with experienced CTA analysts suggested that ACTA techniques can be utilised in context by the explicit recognition and development of socio-cognitive competence /insight. This observation is echoed by other researchers in the field who also note the importance of the widening of NDM and CTA to include important social and organisation-specific areas.

Next, we include two practitioner papers in healthcare by Militello and her teams. First, Militello et al. (2018a) provide insights into the hidden complexities in information flows between primary and specialty care clinics. They describe the macrocognitive challenges faced by consultants at both individual and system levels, and conducted a study of the consultations process in the US Department of Veterans Affairs. Using interviews, observations, and document review, they identify limitations in the current documentation of information flow. Their approach offers a more ecologically-oriented, descriptive model of information flow, and highlights common breakdowns in the consultations process.

Second, Militello et al. (2018b) provide an understanding of how primary care clinicians make sense of chronic pain. In the US and some other countries, the complexities of caring for chronic pain are exacerbated by individual and public health risks associated with commonly used opioid analgesics. To help understand and improve pain care, this article uses the Data Frame theory of sensemaking (Klein et al. 2006) to explore how primary care clinicians in the US manage their patients with chronic noncancer pain. Critical decision method interviews with primary care clinicians identified several patient-related, social/environmental, and clinician factors that influence the frames the clinicians use to assess their patients and determine a pain management plan. Findings suggest significant ambiguity and uncertainty in clinical pain management decision making. To help manage these, interventions to improve pain care could focus on supporting sensemaking in the context of clinical evidence rather than attempting to provide clinicians with decontextualized and/or algorithm-based decision rules. Interventions, therefore, might focus on delivering convenient and easily interpreted patient and social/environmental information in the context of clinical practice guidelines.

Next, a paper by Suss and Ward (2018) documents the perceptual–cognitive expertise in law enforcement. Four studies are described, which investigated experience-based differences in police officer decision making in complex, rapidly unfolding, and uncertain situations. In these naturalistic situations, decision makers must first generate—for themselves—at least one option before intervening or taking action. Hypotheses about option-generation processes were derived from apparently competing theories, skilled decision making, and expert sensemaking. A variety of cognitive task analysis techniques, including experiments using option-generation and temporal-occlusion methods and process tracing measures (e.g., retrospective verbal reports, video-stimulated recall) were then used to test the hypotheses. The authors conclude that the two theoretical approaches appear to be complementary rather than competing. When the situation is relatively familiar, officers can quickly recognize the situation and identify an appropriate response. However, when situations are less familiar, more complex, and/or more uncertain, officers may need to engage in rapid situational assessment and diagnosis so that they can quickly make sense of the situation. The implications for law enforcement officers and for law enforcement training are examined.

An exemplar in engaging with technology is then provided by Parnell et al. (2018) which focuses upon the intentions of drivers to engage with secondary tasks during driving on both the road and in a simulator. They note that while the consequences of distracting tasks are often assessed in settings where the risk of engaging is reduced (i.e., simulators), the drivers’ decision to engage with secondary tasks is often ignored. The study used verbal protocols to provide insights into the drivers’ intentions; this enabled an understanding of when drivers engage with technological distractions, why they choose to do so, as well as how they may go about doing it. Different road types were found to differentially impact the drivers’ intention to engage, as did the types of secondary tasks, with some tasks having an increased willingness to engage compared to others. The decisions that drivers made to engage with secondary tasks in the simulator were found to correlate strongly to their decision to do so on the road. This provides support for the use of simulators when assessing the drivers’ decisions to engage with secondary tasks. The effect of verbal protocols on the drivers’ speed metrics was also assessed to determine if the degree to which the measure interfered with the performance it was assessing.

Imbsweiler et al. (2018) report results of a questionnaire study on cooperative traffic situations which was analysed from a naturalistic decision-making perspective. By means of the NDM approach and the use of recognition-primed decision-making links between planned action and the expected action between road users were identified. It is hoped that the findings will complement design recommendations for automatic vehicle guidance systems in cooperative situation scenarios.

David and Schraagen (2018), analyse the communication dynamics at the transaction level with a detailed case of Air France Flight 447. They argue that both adaptive and maladaptive patterns of adjusting to a situation are rooted in behavior patterns that should be studied in light of their previous history of transactions. Those patterns may develop over longer time scales yet exert their effects during unexpected situations on much shorter time scales. Butts’ (2008) relational event model was used to examine the communication dynamics amongst the pilots, and illustrate how communication patterns may be studied by considering sequences of relational events, thus adopting a dynamic, de-contextualised approach to system analysis at a ‘transaction level’. The analysis of the communication transcript revealed patterned changes in communication dynamics in the cockpit after the flight entered an unexpected situation, which led to the biased strengthening or weakening of certain links in the network. These changes—even though preliminary due to the limited number of agents analysed—suggest that capturing the structural composition of a system at the transaction level assists in explaining how transactions fail, and can be used for the development of better system structures or training procedures for system interaction.

Attfield et al. (2018) provide insights into distributed sensemaking—an area which continues to require further theoretical and practical development. Elaborating upon the data-frame model the authors propose a distributed resources model of sensemaking. It describes resources involved in sensemaking in terms of three domains: knowledge and beliefs, values and goals, and action. Knowledge and beliefs are concerned with how things are; values and goals are concerned with how things are desired to be; and action provides the means for redressing the gap. Central to the model is the idea that these resources can be distributed across a cognitive work system, both actors and representational media. Hence, it aims to provide a framework for analysing sensemaking as distributed cognition.

Harrington et al. (2018) report the findings of a series of interviews with search and rescue volunteers. The purpose of this study was to discover what types of decisions are made during missing incidents; including a consideration of the factors which affect these decisions and the main focuses of attention throughout the incident.

Interviews were conducted using the critical decision method (CDM) to elicit specific information about the decisions and challenges faced by search and rescue teams during missing person searches. The authors created a decision-sequence diagram to show the sequence of each decision and trends across all incidents, which should be of use to other practitioners and researchers in this area.

Our final paper is a short practice note on ‘lessons learned’ from cognitive skills training, Klein et al. (2018), drawn together from a number of ‘ShadowBox’ training intervention projects. The authors explain how the original ShadowBox mission statement has evolved, and recommendations are offered for others who are engaged in cognitive skills training to share good practice.

3 Emerging challenges and future research

Mosier et al. (2018) map some of the areas of future work and future challenges for NDM related work including: expanding the study of the domain of expertise; the importance of recognizing the role of affect and cross-cultural work; extending methods; considering the importance of technology and its design and complex hybrid ecologies (Mosier 2008); sense-making in the hybrid ecology and multi-team systems. This latter area is also echoed by Stanton (2014).

Traditionally, NDM research has focused on decision-making activities ‘in-the-heads’ of the experts. One future direction of research could be to broaden the research foci to include decision-making ‘between-the-heads’ of multiple agents who contribute to the process (Stanton 2014). This would include both human and non-human agents (both artifacts and artificial intelligence). There are several drivers for this research, including (but not limited to), evidence from the field of embodied and distributed cognition (Hutchins 1995), macrocognition (Klien et al. 2003), cognitive systems engineering (Stanton et al. 2017a), and sociotechnical systems theory (Stanton et al. 2019). This perspective regards decision-making as an emergent property of collaborative systems, arising from the interactions between agents, both human and technological. The notion that decision-making is not confined to humans in systems often causes conceptual difficulty, but the idea has good pedigree (Stanton et al. 2017b). This perspective has been applied to most, if not all, cognitive processes. For example, the study of distributed cognition (DCOG: Hutchins 1995) and distributed situation awareness (DSA: Stanton et al. 2017b). DCOG is characterised by multiple ‘agents’ (both human and non-human) working together in pursuit of common goals for which high levels of communication and coordination are required (Stanton 2014; Stanton and Roberts 2018). DCOG is related to the theory of ‘transactional memory’, which showed the reliance that people have on other people and machines to remember for them (Wegner 1986; Sparrow et al. 2011).

Macrocognition has been applied to many fields, most recently in submarine command and control (Roberts and Stanton 2018). The between-the-heads macrocognitive processes (Cooke et al. 2008) have been used to understand how the control room develops and maintains the tactical picture to remain safe. The team detect contacts in the sound room, and check that contact designations are correct before communicating these to the picture room. Contacts require coordination from multiple sensors to manage information exchange between the sound room and the picture room. The control room build the tactical picture to inform planning. Tactical decisions are then informed by interpretation of the tactical picture. The macrocognitive functions of detect, check, communicate, coordinate, plan and decide all occur above the level of individual team members and involve both human and technology (Roberts and Stanton 2018).

Recent reviews of DCOG and DSA have argued that the sociotechnical systems perspective is the most flexible, as it can be applied to micro, meso and macro systems alike (Grote et al. 2014; Stanton et al. 2017b). In sum, we concur with Mosier et al. (2018), and endorse their view that methodologies and theories should adopt a multi-disciplinary approach to understanding the complexities of decision making in the current world and look forward to seeing how sociocognitive (Gore et al. 2018a) and sociotechnical systems perspectives develop.