Collaborative Time-Sensitive Targeting

DRAFT – November 8, 2006

Interruption Recovery Assistance tool for Collaboration in TST Displays
Design of Experiment: Pilot Study

Jordan Wan

Humans and Automation Lab (HAL)

Massachusetts Institute of Technology

Introduction

The goal of this project is to investigate the effectiveness of integrating the Interruption Recovery Assistance tool (IRA) into the Collaborative Time-Sensitive Targeting (TST) interfaces. In particular, the aim is to build on the existing TST Mission Commander Interface, and help the Mission Commander (mCDR) maintain context and support decision making that is line with the priorities of the overall mission.

This study will build upon two pieces ofresearch that were also done at HAL:

1. Assisting Interruption Recovery in Supervisory Control of Multiple UAVS (Scott, Mercier, Cummings, 2006)
2. Collaborative Time Sensitive Targeting (Scott & Cummings, 2006)

In a collaborative, time sensitive targeting operation, the mCDR is responsible for the success of the overall mission. The assets available to the mCDR are the human operators and any technology (interfaces as well as vehicles) at their disposal. The mCDR must balance both human-human as well as human-computer interactions. Thus, it is foreseeable thathe/she will be burdened with the challenges of interruptions and multi-tasking while trying to manage ongoing, emergent problems of the mission.

The goal of the IRA toolis to help the supervisor minimize recovery time and support time-sensitive decision making during the course of the mission.

The IRA will be integrated into the set of existing TST displays that were developed to help the mCDR of a team of UAV operators successfullyprotecta political convoy transiting through an unrevealed and potentially hazardous area of interest.

The Collaborative Time Sensitive Targeting interfaces were developed based on recent work in the Computer-Support Cooperative Work research field (Carroll et al., 2003; Carroll et al., 2006; Millen et al., 2005; Muller et al., 2004). This set of 2 large-screenand 1 tablet displaysisintended to serve two purposes that will help TST team supervisors during a mission:

-to help them understand how the current battlefield situation relates to their operators’ individual activities/performance. This information should help the TST team supervisors understand which operators are effectively handling their current mission tasking and which operators might need assistance or a change in tasking, and

-to help them prioritize the team’s current problems in the context of the overall mission priorities.

Objectives

The overall objective of the IRA is to help the mCDR handle interruptions. Specifically, I want to evaluate the effectiveness of the proposed additions/changes to the current TST displays in a laboratory user experiment.

The main goal of this experiment is to evaluate whether the IRAreduces the negative impacts of interruption in a Collaborative TST environment. After an interruption occurs, the IRA should be the main decision tool used to regain context (recovery) and provide the mCDR with all relevant information to make the decision in-line with the priorities of the mission.

The IRA is constructed to provide a constant stream of information/data that can influence short term decision making. This will allow the mCDR to execute well-informed decisions based on recent events in the mission. Hence, another foreseeable consequence we may observe from the IRA is an improvement in situation and activity awareness for the mCDR during themission. However, this added benefit is a lower priority but can be an added bonus to the addition of IRA in the interface.

Research Questions

To achieve these goals, the following research questions will be investigated in this user study:

1. Does the IRA help minimize the negative impacts of an interruption?

2. Does the new Mission Commander Interface (MCI tablet display) design change the way a user recovers from an interruption?

If so:

What was different about the process?

3. Does the IRA help maintain situation and activity awareness at all times during the mission?[MLC1]

If so:

i.Is this information presented in a form that is useful to decision making?

ii.Is this information presented at an appropriate level of detail?

If not:

i.What information is missing?

ii.What changes can be made in the interface to improve communication of critical information.

iii.Is there too much or superfluous information?

Experimental Task:

The actual experimental task will closely parallel the scenarios created in the Collaborative TST Research (Scott & Cummings, 2006).

General TST Task:

The primary task involves a team of TST operators working together to secure a large geographic area (the team’s area of interest (AOI)) to ensure the safe passage for an important political convoy that will be traveling through the area in the near future. During the task, the team will be required to survey the area for potential threats. Once hostile targets have been identified, the TST team must coordinate with an external strike team to engage these hostile contacts before the convoy is within weapons range.

In order to secure the AOI, the team will be required to utilize a number of highly autonomous Unmanned Aerial Vehicles (UAVs). Various team members will be required to monitor the progress of these UAVs as they provide surveillance of the large AOI and to reroute the UAVs from their original surveillance course, as necessary to secure the area.

The TST team consists of three UAV operators, each responsible for controlling 3 UAVs and one mission commander overseeing the team’s mission progress (including monitoring the performance of the 3 operators). In particular, the UAV operators are each responsible for updating the mCDR with the status of their assigned UAVs as well as any surveillance information they come across in their respective AOIs. When a foreign target emerges in an operator’s AOI, it is their responsibility to notify the commander and perform a secondary task (which simulates the detection latency time of whether the target is hazardous). If the operator detects a hazardous target, they must communicate the location of the target to the strike team to eliminate it before the convoy enters weapons range[jw2]. The mCDR is responsible for ensuring the safety of the convoy and for managing the workload of the UAV operators on his team throughout their mission. To achieve these mission objectives, the mission commander can make three types of strategic decisions.

1. Requesting the convoy hold its current position if its intended path is not deemed safe for passage
2. Requesting supplementary surveillance data from a nearby multi-sensory aircraft acting as the local airborne warning and control system (JSTARS).
3. Reroute unused UAVs to complete the mission of any UAVs that were destroyed by targets.

While there are many collaborative components to this task, this project will focus on the performance of the mCDR in his/her decision-making and managing the overall tasking of the TST team. In our study, the activityof the individual UAVsand UAV operators will be simulated, along with the rest of the battlefield environment.. This will represent a situation where a commander is overseeing a set of remote team members, which is a very realistic scenario in network centric operations.

Mission commanders will be able to make unlimited use of the first type of command decision (holding or resuming the convoy) during a mission. However, they will have only limited access to JSTARSsurveillance data. They are only allowed to request this supplementary surveillance data up to three times per mission.

Themission commander decision controls are found on the mobile tabletPC interface that participants will be provided during the experimental sessions. Participants will also be asked to use the tabletPC interface to record each occurrence of a target being scheduled for a strike within 30 seconds of the start of its threat envelop (i.e., the period of time that the target is within weapons range of the convoy, based on its location and its weapons capability). For example, if a target is schedule to be destroyed at 5 seconds before the convoy will be within range of its weapons, the participant should record the ID of this target as soon as they notice this tight-scheduling situation. This subtask will be referred to as target buffer violation detection. Finally, the IRA will also be located on this display.

The next 2 sections will go into detail the usability of the 3 displays. First, the general displays (as created by the Collaborative TST project) will be detailed, followed by the IRA component which is proposed by this project.

TST Displays

Figure 1 (below) shows the two team supervision displays, which include a variety of mission and activity awareness information that is dynamically updated as the mission progresses. For example, in the Map Display Interface (Figure 1a), the color of the UAVs (the symbols) represents their current status (blue: surveillance, orange: operator confirming possible target, grey: last known location, possibly destroyed). The color of the targets (thesymbols) represents the team’s current progress on prosecuting that target (red: identified, grey: destroyed). The color of the sub-AOI boundary indicates the current performance status of the corresponding operator (black: nominal, yellow: potential threat to convoy based on insufficient surveillance, red: operator likely overloaded and convoy safety currently or soon to be in jeopardy[MLC3]).Additional information on the spatial map include target range rings (indicates the potential weapons range of a detected target); UAV path filter (shows the path in which each UAV is projected to follow). The Threat Summary and Strike Schedule panel at the bottom of the display show a) when the convoy will be within the firing range of an undetected region (Potential Threats), b) when the convoy will be within firing range of a detected target (Known Threats), and c) when the detected targets will be scheduled to be eliminated by the external strike team (Strike Schedule).

In the Mission Status display, various mission and operator activity information are provided in relation to the convoy’s current and expected level of safety. TheMission Status displays the current strike plan in the context of the current and expected convoy threat level. This display also provides the team supervisor information about the tasking performance of the individual UAV operators, including indications of the convoy’s threat uncertainty based on insufficient UAV surveillance in each operator region, and summaries of operator surveillance performance levels. This display also provides current communication connectivity status to external resources like the convoy, the AWACS and the strike team.

(a)


(b)
Figure 1. The Team Supervision large-screen displays: (a) the Map Display interface and (b) the Mission Status Summary interface.

The 2 displays shown in Figure 1 will be displayed on large, wall screen displays in the Humans and Automation Lab team testing environment. The team supervisor will indicate their decisions throughout the experimental sessions on a button panel provided on a mobile tabletPC, shown in Figure 2. These decisions will be input into the simulation, changing the situation as appropriate. The bottom half of the Mission Commander Interface (Decision Controls) follow the same functionality as before (See reference: Scott & Cummings, 2006). The only difference in this case is that the locations of the controls have been modified to allow the integration of the IRA into the tablet screen. The functionalities of the IRA tool will be explain in the next section.

Figure 2. The Mission Commander Interface: The top displays IRA; the middle allows for command decisions, inputting threat buffer violations; the bottom displays Message History (a log of events[MLC4]).

IRA Display

The IRA interface is based on the previous research approach of interactive bookmarks (See paper: Scott, Mercier, Cummings, 2006). The IRA has two fundamental design goals:

1)To build on the iterative bookmark concept, but try toallow more user interaction with the bookmark itself.

2)To mitigate the disadvantages of the previous research by eliminating the separation of the replay interface with the main screen (ie. incorporate IRA directly into the main interfaces).

The IRA will support 4 events:

Event / Action to Take
1. Late Strike / Indicateoccurrence on the interactive bookmarktimeline with an icon. If user clicks on the event icon, circle the corresponding target and its threat envelope on Map Display Strike Schedule.
2. Convoy is hit / Indicate occurrence on the interactive bookmark timeline with an icon. If user clicks on the event icon, display a vertical red bar on the paththe Spatial Map in the Map Display Interface.
3. UAV shot down / Indicate occurrence on interactive bookmark timeline with an icon, If user clicks on the event icon, display a red X on the Spatial Map in the Map Display Interface.
4. Loss of communications / Indicate occurrence on interactive bookmark timeline. If user clicks on the event icon, show corresponding link on the same timeline with a connecting arrow.

In line with the first design goal, when events 1, 2 and 3 occur, the user can click on the icon on the iterative bookmark, and a corresponding visual cue will be displayed on the map display interface. This allows the user to instantly get a visual cue as to where and when on the spatial map did the event occur with respect to the current time and location of the convoy. After 10 seconds, the visual cue will fade away.

When these four events occur, IRA will automatically update its timeline and display the icons corresponding to the time of occurrence. [MLC5]Clicking on one of the first three events will reveal further details on the map display (see below). The figures below illustrates a screenshot of a realistic situation during a mission and what happens if you clicked on one of the first three timelines[MLC6].

Experiment Design

The experiment will be a 2(Assistance Type) x 2(Decision Difficulty) mixed design, with repeated measures on the Decision Difficulty factor and between subjects on the Assistance Type factor.

Decision Difficulty (DD). The two levels of DD are simple and complex. The difficulty relates to the complexity of decision(s) that need to be made when participants return from interruptions. In the case of a simple DD, only a single decision needs to be made to address the current situation. With a complex DD, several decisions are possible to address the immediate situation, but based on prior events there is one decision that is most in line with mission priorities.

An example of a simple decision is the case of a convoy approaching or entering an unsurveilled area due to surveillance hold-ups occurring while the mCDR was gone. Upon the mCDR’s return, the correct response would be to either request JSTARS surveillance data or hold the convoy while UAVs are still surveilling. If we disabled JSTARS while mCDR was gone, the only correct response would then be to hold the convoy (since the mCDR should notice that JSTAR is no longer available).

An example of a complex decision is that while the mCDR was gone, one of the UAVs was destroyed and thus needs to be reassigned. If the mCDR can correctly identify the best UAV to reassign to this within a reasonable amount of time upon returning from their interruption or alternatively holding the convoy or using JSTARs, we could measure if they identified the optimal strategy for that scenerio.

Assistance Type (AT). The two levels of Assistance Type factor are: without IRA (original TST Displays), and with IRA (new Mission Commander Interface with IRA integrated).

Participants

12 participants will be recruited to participate in this study. Participation will be strictly voluntary. If possible, participants with military supervisory experience will be recruited. The experimenter will explain the study to the participants and answer any questions about the study they may have. Participants will be required to sign an Informed Consent Form before performing any of the experimental activities.

Procedure

The intended procedure is as follows. This may change slightly, based on pilot testing.

Participants will each perform the experiment individually. They will first be welcomed and provided a brief introduction to the experiment. They will then be asked to sign informed consent forms and complete a background questionnaire gathering demographic information and their previous experiences with computers, the military, gaming, and team supervision.

Next, the experimental task will be explained in detail, along with the experimental software and interaction environment (30-45mins). They will then complete a series of short practice trials (likely three 5-min trials, but may changewith pilot testing, ~15-20mins). Participants will then complete two experimental trials, one for each level of Decision Difficulty, with order of presentation counterbalanced across participants (15-20mins per trial, 30-40mins total).

Once the experimental trials have been completed, participants will be asked to complete the Cummings-Myers Display Quality Rating Scalefor the IRA tool only(Cummings et al., 2006, also see Appendix B). Participants will then partake in a semi-structured interview with the experimenter, which will inquire about their decision making strategies while reviewing a screen replay of their final experimental trial (Appendix A). Participants will then be debriefed about the experiment, paid a nominal fee, and thanked for their participation.

Data Collection