Home

 

Performance-Based Auditing/Assessment

By: John H. Johnson, J-E-T-S Consultants

Charlotte, NC Phone: (704) 987-9858

Note: This paper originally presented at the 2nd ASQ National Audit Conference

It is appropriate that this paper is presented under the Track of "Auditing in a Regulated Environment," because of the lessons learned and pitfalls that can be encountered when the environment regulates the audit process. The U.S. commercial nuclear power industry is probably the most regulated industry in the world; indeed our government oversight group is called the Nuclear Regulatory Commission. In the ultimate (but improbable) adverse condition, a nuclear power plant has a potential for significant safety risk to the public. The NRC's duty to ensure public safety and minimize risk in the design, construction, and operation of nuclear power plants is therefore justified. The regulations that have been developed have had public safety as the basis for required actions.

I. Background

Independent, planned, and periodic audits have been required within the nuclear power industry since the issuance of 10 CFR 50, Appendix B, the Quality Assurance requirements for nuclear facilities. The problem with auditing to meet the requirements of regulations is basic and proven by past experience. Simply meeting requirements will not ensure success, efficiency, worker safety, or the currently progressive term of excellence. Many audits have been performed to determine if minimum requirements were met. The resulting feedback to management from such an audit could be misleading - no deficiencies were identified, no requirements were violated. Management could therefore think that the activity was performed successfully, when in actuality it might have been performed in a manner that was inefficient and unsafe to the workers.

Consider the purpose of an audit. An audit is conducted to provide feedback to management. The purpose of an audit is not to simply meet a specified regulatory commitment. Audits are required by regulation to ensure that regular and formal feedback is provided to management regarding the effectiveness of various programs. Management can then act on the audit feedback to improve the process. The feedback to management from the audit must therefore be technically correct, useful, and directed toward improvement. This requires a substantial departure from auditing to only determine compliance to regulations. If an organization is only interested in regulatory compliance, a minimal compliance syndrome will usually pervade management. With the highly regulated environment at nuclear power plants, this syndrome has caused many difficulties. The minimal compliance syndrome is usually evidenced with supervisors and managers making comments like "where does it say I have to do that?" or "where is the requirement for that?" Since regulations are so prescriptive, the attitude is that meeting them will ensure success. The regulations, remember, are based on ensuring public safety, not on doing all jobs the best way possible.

In organizations with a minimal compliance syndrome, audits are usually conducted to determine if regulations are met. In a nuclear power plant, this means that audits will be limited to Safety Related activities, because those are the ones that are most regulated by the NRC. Safety Related activities are those that could prevent or mitigate the consequences of an accident. Generally, this includes items that impact the safe shutdown of the reactor and maintenance of the reactor in a safe shutdown condition. The portions of the plant designated as Non-Safety Related are not subject to a QA audit or surveillance program. These parts of the plant may include the turbine/generator, feedwater, compressed air, switchyard, and electrical distribution systems. With these systems given little management attention for improvement, it is no surprise that these plants have frequent unplanned outages that originate in Non-Safety Related systems. Plant management then struggles to find what causes their facility to be so unreliable.

II. The Performance-Based Approach

What is the remedy to this situation? One answer is a redirection of management attitude. Compliance with regulation must be understood as a minimum effort that is the first rung in a ladder that leads to success. Success at a nuclear power plant can be defined by what can cause the plant to be shut down unexpectedly. Two conditions may cause this. The first is equipment failure that forces shutdown until the equipment is returned to operability. The second condition is when the plant is outside its allowable safety parameters and may present an increased hazard if operation is continued. In simple terms, the two terms that determine plant operational performance are reliability and safety. "Good" plants operate more safely and reliably than "bad" ones.

If the main performance criteria for nuclear plant operation are therefore reliable and safe operations, management must focus attention on the activities that ensure safe, reliable performance. Since these are the primary parameters for successful performance, we use the term Performance-Based to identify the management style that provides emphasis in these critical areas. The attitude of "if it's not Safety Related, it's not important" is not compatible with a performance-based philosophy. Performance-based management emphasizes successful completion and on-going improvement of all activities that impact plant safety and reliability. Plant management must understand, communicate to workers, and enforce a philosophy that ensures that the facility is constructed, maintained, and operated in a manner that is favorable to long-term reliability and safety. The term safety will probably also require some redefinition. The NRC term safety generally means public safety. Most companies should define safety as "the relative freedom from harm or hazard to workers and the public." Since about 70% of unplanned plant shutdowns originate in systems designated as non- safety related, it makes sense that more attention is given to ensure their safe and reliable performance.

The purpose of a performance-based nuclear audit is thus to provide correct and useful feedback to management that supports continual improvement in plant safety and reliability. This is very different from an audit that determines if an activity meets regulatory mini- mums. A regulatory compliance audit simply states "here is the regulation; what you did was contrary to..." A performance based audit must have the three aspects listed above:

  1. Technically Correct
  2. Useful to Management
  3. Emphasize Safety and Reliability Improvement

Each of these will be discussed in order.

1. Technically Correct. The information in the audit must be based on the technical basis for the requirement, not the requirement itself. For example, a technical specification requirement might be that a compressed air system be operated between 95 and 105 PSI. If an auditor conducting field observation during an audit finds a gauge that indicates the system operating at 90 PSI, it is not enough to state the above requirement, then say "contrary to the above, the system was operating at 90 PSI." This does not address the real technical problem. Consider the questions that management must answer to resolve this indication:

bulletWas the gauge reading correctly?
bulletWhat was the operating status of the equipment connected to the compressed air?
bulletIf the pressure was 90 PSI, what allowed or caused it to be low?
bulletWhat is the impact of this condition on plant operability, safety, and reliability?

A technically correct statement would answer these questions. It would also provide the impact that low pressure would have on the safe and reliable operation of the equipment that used this compressed air source.

Note that the auditor has identified a real problem, in-process, that can be corrected. This problem was not found after the fact through a documentation review, or after the problem caused an equipment failure. An important aspect of the technical value of performance- based auditing is then to find and correct problems when and where they occur. This emphasizes the importance of evaluating activities in-process. Observing activities in-process has become the benchmark of a performance-based auditing program. Industry experience has proven, however, that the effectiveness of the audit is not directly related to the amount of field observation. It is more related to what is observed and the attitude and knowledge of the observer.

2. Useful to Management. This follows the logic of the above paragraph. If the auditor just provides the requirement and the deviation, management is left with too many unanswered questions to make effective and timely corrective action. Management must retrace the auditor's footsteps and answer the same questions that the auditor should have provided initially. Another item that is useful to management is an attribute that is not currently deficient, but could become so if left to degrade. In a minimum compliance situation, these are usually ignored because they do not exceed regulatory requirements. Management therefore only finds out there are problems when the item or activity reaches a point of unacceptability; that point is obviously too late. Useful feedback to management includes identification of issues that not only are problems, but ones that may lead to problems. Management can then take action before the situation reaches a point of unacceptability. Line management should be considered the auditor's customer. All aspects of the audit should be directed to provide the customer with information that can be used for improvement.

3. Emphasize Safety and Reliability Improvement. As stated above, problems that could lead to deficiencies in the future should be identified so management can take action. Beyond that is the scope of activities subjected to audit. This may appear to be common sense, but in a performance-based system, all activities that impact the safe and reliable operation of the facility should be subjected to audit. This includes not only safety related activities, but non-safety related activities that impact plant reliability, or worker safety. In these areas, there is even more opportunity for improvement than in the safety- related activities. Since non-safety related activities are less regulated. They are allowed more freedom for innovation and efficiency improvement. The auditor should be a tool for management to use to stimulate and encourage innovation and improvement.

An auditor evaluating a safety related activity must be aware of not only the requirements, but the intent behind the requirements. The auditor should not just determine if the procedure was followed, but if the work was performed to achieve the intended result.

Differences

It should be apparent by now that a performance-based audit will differ from what has been the normal nuclear QA audit of the past. A comparison might be that listed below:

Attribute Traditional Performance-Based
Audit Focus Review of documents Important activity evaluation
Goal Regulatory compliance Feedback to improve process
Methods Review documents,

Interview managers

Observe actual activities, Seek innovation at line level
Report Describes noncompliance's Evaluate to improve process

III. What it Takes

The performance-based approach has only been proven to work when certain prerequisites are met. Three fundamental prerequisites are involved. The first and most important is upper management support for the concept and its implementation. The second is ensuring technical competence for the evaluators. The third is education of line personnel on the intent and refocus of the audit program. Each of these will be discussed individually.

Management Support

Top management must support any quality improvement program in order for success to have a reasonable chance. Although some have touted the idea of "bottoms up" improvement programs, they simply have not proven to be successful in the long term. Jeff Beardsley was right when he said the three things that cause quality improvement programs to fail are first, a lack of management support; second, a lack of management support, and third... - I think you get the idea. In a performance-based program, auditors look at different things and they look at things differently. Although the previous sentence may sound like word games, it is nonetheless true. Auditors doing a performance-based audit are interested in the in-process activities that may impact the important performance parameters of the area being audited. For nuclear power plants, that means they will be interested in any activity that impacts the safe and reliable operation of the plant. Since many past audit programs have been limited to safety related activities, this represents a major change. There are many activities that impact plant reliability that are not designated safety related. When the audit program first starts to delve into these areas that were previously off-limits, there will be a natural resistance from line management of those activities. The attitude of "You never looked at us before, why now?" is to be expected.

Top management must endorse and support the direction of a performance-based audit program. Managers must anticipate the resistance from line organizations and prepare them for the new audit process. A teamwork philosophy of identifying opportunities for improvement throughout the organizations has been successful here. Upper management must demonstrate commitment to the process and not give in the first time a line supervisor complains about being subjected to audit. Management must continue to demonstrate commitment by attending and actively participating in close-out meetings and corrective action.

Technically Qualified

In order for the performance-based audit program to be technically correct and provide useful feedback, it must be conducted by technically qualified personnel. There has been an idea that once someone is qualified as a lead auditor, they may effectively audit any- thing. This will not work for performance-based audits. The performance-based auditor must have an understanding of the basic process that is being evaluated. Understanding the intent and objectives of the process or activity allows the auditor to identify opportunities for improvement that are technically credible. If discrepancies are noted, the auditor will be able to justify and explain to line management the impact on safety and reliability if the problem is not corrected.

In reality, there are some audits that will be performed by auditors who are not technically qualified in the activity or process being evaluated. This presents a challenge to achieve the objectives of a performance-based audit. The answer to this problem is to seek and find expertise from elsewhere. Outside technical expertise is available from consultants, but it is expensive. Another source is the group being audited, or a similar group. This may reap several benefits. One is it fosters the teamwork approach. Another is the appreciation of the audit process that returns to the line organization after participating in the audit. A third is the technical value to the audit that is added by someone understanding the process as only someone who has done it can. The cautions here are obvious - do not allow people to review their own work, and be aware of friendships, traditions, and hidden agendas.

Line Personnel Education

The line organization needs to understand why the audit process is being changed. Management must realize that all changes will be resisted at first. Management and auditors must prepare auditees for the new approach. The purpose of the performance-based approach must be explained before implementation and again at each entrance meeting. This will be particularly true for vendor audits. Vendors do not usually expect a performance-based approach, and may be surprised when faced with it the first few times. The auditor must prepare or, if necessary, formally train the auditee in the purpose and methodology of a performance-based audit. The performance-based approach should always avoid surprising the auditee. Teamwork and communication are key to identifying and implementing improvement.

IV. Summary

Performance-based auditing is the logical progression for auditing beyond regulatory compliance. In the highly regulated nuclear power industry, we have learned that regulation can ensure public safety, but cannot ensure success in plant operation. Audit programs that stop at determining the level of regulatory compliance or are limited to safety related systems cannot provide management with the information needed to improve. Management should be able to use the audit program for its eyes and ears on the effectiveness of activities. Many past audit programs have not provided very good vision or hearing for management. On the ladder to excellence, compliance with regulations is only the first rung. A performance-based audit program places audit resources where they can best be used. It can give management useful and timely feedback that can be effectively used to improve overall performance.

The author welcomes comments on this article!  Send comments to:

john@jetsquality.com

Note: This article is for your personal use only. Do not reproduce, revise, or distribute this article without permission of J-E-T-S!

Home ] Up ]

J-E-T-S celebrates over 30 years of service excellence in 2017!               Phone:  704-987-9858
Send mail to webmaster@jetsquality.com with questions or comments about this web site.
Copyright 1984-2017 
Last modified: December 05, 2016