TEM Data: NOSS data collection is guided by, though not limited to, TEM. TEM is a conceptual framework that aides, from an operational perspective, in understanding the interrelationship between safety, human performance, and the dynamic and challenging contexts in which ATM duties are performed. The TEM data yielded by NOSS is a mixture of quantitative (TEM summaries) and qualitative data (rich textual descriptions). NOSS provides detailed information on the threats and errors that occur during operations, how they are managed, and the impact they have on operations. See all Case Studies.
Summary level TEM data with Benchmarking: The quantitative information obtained by NOSS yields summaries of various TEM metrics. These summaries can provide a “big picture” perspective on what is occurring in operations. When working with the NOSS Collaborative, these quantitative summaries will include de-identified comparisons with similar NOSS Collaborative facilities, which help when interpreting findings. Benchmarking can help identify potential areas of strength or vulnerability, and thus serve as reinforcement of operational strengths or as motivation to address possible vulnerabilities. For example, a particular sort of threat or error may be observed during 20% of observations. Benchmarking with other NOSS Archive facilities may indicate that 20% is very low (or high) compared to other facilities, which may affect the manner in which the ANSP interprets the result. See all Case Studies.
Work-as-done: Quantitative data may help identify areas of strength or vulnerability, but it does not provide details as to why things are occurring. In addition to quantitative summaries of TEM metrics, NOSS provides rich descriptions of all the TEM-based events and other elements of interest that are occurring in operations. This contextual information provides details of the operational environment and how controllers perform their work duties within that environment. The combined quantitative and qualitative information provides a good indication of what is occurring in daily operations and provides a lot of information as to why things are occurring. See Case Studies 2-5, 7-8, & 11-12.
Vulnerabilities and “what’s going right”: Unlike traditional sources of safety data collection which are triggered by abnormal events, NOSS data is collected during normal operations, prior to failures. This means that NOSS captures two distinct types of safety information – things that are going well (see Case Studies 2, 4, 7, 8 & 11) and areas where improvements could be made (see other Case Studies). Potential vulnerabilities can be addressed before they lead to incidents, while strengths can be identified – at the unit-level or individual-level – so that they may be reinforced or spread to other units and controllers.
Safety intelligence to support decisions: Managers have reported that NOSS has been useful to document issues they may already have been aware of through informal sources such as “gut feelings” or anecdotal reports from front-line controllers. By substantiating informal sources of information with hard data, managers have reported that they are in a better position to justify their actions. For example, one ANSP had a large number of coordination breakdowns relative to other ANSPs. As a result, management made a concerted effort to try to reduce the amount of manual coordination that was required (through equipment acquisition and enhancing LOAs). Subsequent NOSS reporting revealed that coordination breakdowns were reduced by over 75%. Other examples include supporting the need to initiate airspace reviews/changes and to install special lighting and noise reduction equipment in operations rooms. See all Case Studies.
In theory, the ATM system is designed to function safely through the equipment the controllers are given to use -- through the training that teaches them how to use the system and the procedures that provide guidance as to how to operate the system. In reality, the ATM system is full of complex, dynamic interactions and constant changes that cannot be fully anticipated in the system design stages. In short, NOSS is analogous to a ‘health check’ of operations – it provides information on how the ATM system actually is behaving as opposed to how it was intended to behave by using specially trained observers to conduct anonymous, de-identified, non-punitive observations of controllers working positions under normal conditions (e.g. no checks, OJTI, etc).
NOSS is an observational tool: A group of controllers are selected and trained as observers in the TEM framework and the observational methodology. They are then assigned to observe and and record specific elements of the operating environment and their impact on controller behavior. Observers code the threats, errors and undesired states that they observe, as well as the controller’s response, and the outcome. These controllers also write a narrative that provides important contextual information. The data from all observations are aggregated into one database for analysis, the aim of which is to diagnose strengths and vulnerabilities in the facility’s operations.
Conventional safety data sources mainly provide information from abnormalities (incidents, failures, etc), and provide scant information on what is occurring in the absence of abnormalities. NOSS captures data about what is occurring in routine operations, and thus provides data about ATM system and human performance that is not otherwise available. As such, it complements the traditional sources of data for a safety management system (SMS). NOSS is regarded as a core component of an ATM SMS (i.e. it is an ICAO-endorsed method of fulfilling the requirement to monitor safety in normal operations).
Taken together, incident data and NOSS data can provide the most complete picture as to why safety events occur. While incident and reporting programs identify that an event occurred and provide insight into why it occurred in that particular instance, NOSS can augment understanding of system performance by providing base-rate information about normal system and human performance. In other words, the abnormal event (incident), might not have been so abnormal – the systemic issues and human performance patterns that led to it may be more widespread.
NOSS data is collected during normal operations - prior to failures. This means that NOSS also captures abundant information on things that are going well in operations. Strengths – whether identified at the individual or unit level can be useful information. Individual controller techniques or effective unit procedures can be communicated to other controllers and units, and broad areas of strength can serve as reinforcement to staff and aide managers in prioritizing their safety efforts
NOSS should be seen as a system-based tool for the SMS rather than merely human performance based tool. Human performance based tools are heavily tilted towards training interventions and Quality Assurance/Quality Control regimes (and perhaps procedural adaptations to a lesser extent). NOSS data has been applied extensively along those avenues as well. But systems-based data opens up additional “layers of defense” for intervention. This gives managers more tools to make improvements and thus place controllers in a better position to discharge their duties.
NOSS data is collected from the perspective of operational control positions, but the data often reflects strengths and weaknesses in other aspects of the larger ATM system. For example:
Procedures: A NOSS will provide data about procedural drift, and may identify areas in which procedural guidance is ambiguous, contradictory, poorly understood or a poor fit for the operational environment. See Case Studies 2-5 & 11-12.
Training: A NOSS provides information that can be useful for training in several ways. First, it can provide feedback about the fit between what is trained and what is enacted in operations. Large discrepancies may indicate that the training material does not fit the operational environment, or that training concepts are not being transferred to everyday behaviors. Second, NOSS data can be used to direct training programs – praise & reinforce what is working well, and focus on areas where deficiencies were identified. See Case Studies 3, 5, 7 & 8.
Equipment / Workspace: Many ANSPs are introducing new systems and equipment. It is not always possible to foresee how the changes will play out in daily operations. Sometimes, the equipment doesn’t work as anticipated or controllers interact with the equipment in unanticipated ways. NOSS provides information about the benefits or threats introduced by a wide range of equipment, and provides insight into how controllers interact with the equipment. See Case Studies 2-5 & 11.
Airspace / Airport Issues: NOSS data can help to document and better understand shortcomings in airspace design or airport layout. For example, information captured by NOSS played a significant role in redesigning airspace, the addition of parallel, segregated airways, and modifications to aerodrome signage. See Case Studies 1, 6, 9 & 10.
Interactions with Other Aviation System Actors: NOSS captures data that illuminates interactions with other aviation system actors including airspace users, airport operators, the military, and adjacent ATM units. Objective TEM-based data from an ICAO endorsed process can aide when presenting cases to address safety issues to outside parties. In addition to the examples cited in the airspace / airport section above, ANSPs / Units have shared selected NOSS findings with airlines, airport operators and regulators to help demonstrate the need for safety actions. See Case Studies 6, 9, 10 & 12.
Understanding Controllers Shortcuts / Workarounds: As a result of experience, controllers develop shortcuts and workarounds to save time and work more efficiently. These shortcuts frequently involve contraventions of procedures, and are seldom seen during checks/audits, where performance is typically “by the book.” Through a trusted process such as NOSS, it is possible to observe such shortcuts and workarounds. Some may be deemed effective and can be communicated to others within the organization as a “better way of doing things.” Shortcuts and workarounds that have shortcomings in their safety assumptions can also be identified and addressed. See Case Studies 2, 5 & 11.
The following provides an outline of the NOSS process and required resources when working with the NOSS Collaborative to facilitate NOSS:
1. Planning & Preparation: Decide scope of project, education, observer selection, potential discussions with controllers association
1. ANSP Responsibility: NOSS project coordinator or steering committee part time to serve as liaison and guide decision making
2. NOSS Collaborative Responsibility: Provide guidance
2. Observer Training & Initial Observations
1. ANSP Responsibility: Make observers available for 3-4 days for training, initial observations & one-on-one coaching session with NOSS Collaborative facilitator
2. NOSS Collaborative Responsibility: Conduct observer training & observer coaching sessions
3. Data Collection
1. ANSP Responsibility: Observer availability for 3-5 shifts over a 2-3 month period to conduct their allotted number of observations
2. NOSS Collaborative Responsibility: Collate data, monitor observation quality and progress, provide regular feedback to observers & project manager about progress, build database, review and prepare data for data verification
4. Data Verification
1. ANSP Responsibility: Provide operational experts to review data (1-3 days); just show up and participate in guided discussion of NOSS data
2. NOSS Collaborative Responsibility: Facilitate data verification, make changes to database
5. Data Analysis & Report Generation
1. ANSP Responsibility: Nil
2. NOSS Collaborative Responsibility: Analyze data and produce report; detailed summary of trends and issues identified in the data using a mixed quantitative and qualitative approach; de-identified comparisons to comparable NOSS Archive Units
6. Report Delivery & Safety Change Process
1. ANSP Responsibility: Units / ANSPs are in full control of the safety change process
2. NOSS Collaborative Responsibility: Provide the NOSS Report and Raw Data; provide insight into what other Units / ANSPs have done; provide support to ANSP
7. Re-measure: Follow up NOSS 3-4 years later
NOSS is a periodic or cyclical program that is designed to be conducted every 3-4 years rather than on a continual basis. The primary reason for such a lag between NOSS projects is that some interventions, such as airspace re-design or specific training programs may take time to enact. If a follow-up NOSS were conducted too soon after an initial NOSS, it might not be possible to assess all change efforts enacted as a result of the first NOSS. A secondary benefit of having periodic (as opposed to continuous) NOSSs (other than resources) is that the ‘end-point’ of delivering the NOSS report serves as a catalyst to enact safety changes, rather than an endless process of data collection with no natural point for reaction to the data that has been accumulated.
It has recently be noted by some ANSPs / Units who undergo frequent, significant changes that more frequent NOSS projects might be appropriate for their operations – in the order of every other year. The NOSS Collaborative would still caution against a “continuous NOSS” as it would be lacking the traditional end point that initiates the safety change process – in other words, it would be easy to fall into the trap of continually collecting data without acting on it.
Copyright © The NOSS Collaborative. All rights reserved.