EH-89-9 Technical Software Quality Assurance Issues
                         ENVIRONMENT, SAFETY & HEALTH

                                   BULLETIN

Assistant Secretary for                              U.S. Department of Energy
Environment, Safety, & Health                        Washington, D.C. 20585

DOE/EH-0121                     Issue No. 89-9                   December 1989

Technical Software Quality Assurance Issues

I.   Statement of Purpose

The applications of scientific and technical software throughout the DOE
complex are far-reaching and diverse.  Scientific and technical codes are used
in the design, control and monitoring of major nuclear, physical, and chemical
processes, many of which pose potential hazards.  Technical codes are also
used to model, analyze, forecast, and calculate data vital to DOE missions
and/or the above processes.  Inadequate software quality assurance (SQA) for
scientific and technical codes at any phase in their "life cycle" may not only
result in lost time and/or excessive project costs, but may also endanger
equipment and public or occupational sectors. Existing SQA programs from
within DOE to its various vendors have been disparate, ranging from no formal
SQA controls to comprehensive SQA policies, requirements, and implementing
procedures.  The controls established by NQA-1 apply specifically to computer
hardware and although useful perhaps as guidelines, are not adequate for
controlling computer software.  NQA-2, which does address software quality
assurance, is only in the final stages of the concurrence process.  While it
awaits formal promulgation by HQ, all levels of the Department should be aware
of the dangers associated with inadequate scientific and technical life-cycle
SQA controls and, at the same time, of the wealth of relevant policy and
procedural documents and existing infrastructure that can be utilized in
establishing such controls at a given facility.

The purpose of this Bulletin is to give the reader an appreciation of the
magnitude of the problem and, thus, the need to establish SQA programs by
presenting selected DOE occurrences where inadequate controls are clearly a
root cause.

II.  Unusual Occurrences Where Inadequate SQA Is Identified As A Root Cause

The following is a brief review of selected unusual occurrences associated
with inadequate SQA.

-    Software which controlled pre-heat and pre-cool coils in the makeup air
     unit located in the equipment penthouse did not call for heat when
     temperatures dropped below freezing.  After two nights of below freezing
     temperatures, the coils in the make-up air unit froze and burst.  The
     resultant flooding caused $4,216.00 worth of damage to an AT&T fiber
     optics systems and $45,900.00 to two IBM computers.

-    A gantry robot was given a command to pick up a 20" x 40" container, a
     size for which it had not been pre-programmed, and move it to a slot in
     the "round" room (space dedicated for storage of nuclear material).  When
     the robot failed to lift the container high enough to clear the conveyer

     upon which it was resting, it tilted, and the system operator hit the
     emergency stop button.  He then released the container from the robot's
     grip using a manual operation mode and placed the system into a "Park"
     status, from which he expected normal operation would resume.  Instead,
     when the operator issued a "store the container" order, the robot crashed
     into an adjacent shelf.  As a result, the robot was removed from service.

-    Software developed to calculate nanocuries per gram (nCi/g) of nuclear
     waste was improperly coded.  During operational testing of the code, mass
     quantity calculations were evaluated but nanocuries-per-gram calculations
     were not.  As a result sixteen, fifty-five gallon drums were buried as
     low-level waste, when in fact they may have been TRU.  The drums had to
     be identified, located, exhumed and reburied at an estimated cost of
     $460,000.00.

-    Routine hand calculations to verify the results of computer generated
     piping stress revealed that the commercially designed, developed and
     preverified software package for calculating pipe stresses, had omitted
     the gravitational components of occasional piping stress from the codes.
     The manufacturer was notified and subsequent Errata were published and
     distributed to those users who had purchased the NQA-1 quality assurance
     option.  The omission of this data could have significant impact on the
     potential for process-piping failures.

UORs have been submitted by DOE contractors to report only those occurrences
resulting in injuries and/or property loss.  Instances where poor SQA controls
have resulted in occurrences of a strictly programmatic nature (e.g. milestone
slippage, program inefficiencies) are not identified. These can however, have
significant product/program impact.

III. Verification & Validation

The life-cycle phases of software are traditionally identified as:
conceptualization, specification, preparation, verification and validation,
implementation, maintenance and decommissioning.  Documentation is required
for each of these phases.  Although it is absolutely imperative that an SQA
program involve the entire life-cycle of the software, for the purpose of this
Bulletin, the verification and validation phases will be emphasized.

The ASME NQA-2, Draft part 2.7, defines Verification and Validation activities
as those that:

     a.   ensure that the software adequately and correctly performs all
          intended functions; and

     b.   ensure that the software does not perform any unintended function
          that either by itself or in combination with other functions can
          degrade the entire system.

These activities are performed for each system configuration which may impact
the software.  Testing is the primary method of software validation.
Calculations may be performed by hand or by utilizing other independent
preverified computer programs.  As changes and modifications are made in the
field, to meet site-specific needs, the verification and validation testing
activities may not always be performed rigorously.  It is tempting to
sacrifice the testing and documenting activities for expediency.  However, the
need for safety and the assurance that the software only performs the

functions for which it was intended and furthermore, does not perform any
unintended function, overrides any short term expediencies.  It is noteworthy
that in the long run the cost of controls placed at each software life-cycle
phase is less than that for rewriting and reconfiguring the preceding phase.

Changes, modifications and corrections need to be documented, assessed for
impact on past and present applications of the software and the results
provided to affected organizations.  Administrative controls need to be
established and implemented requiring a formal procedure for 1) determining
change and modification procedures and 2) reporting software errors and
failures.  The point is to assure that changes are successful as defined by
NQA-2, Part 2.7, and problems promptly reported to affected organizations.

IV.  Recommendations:

The essence of SQA is to prevent problems, to remove defects as they are
found, and to contribute to the usability and maintainability of the software
(Fujii 1978).  It is vital that safety-critical software be thoroughly
monitored throughout the software's life-cycle, as well as when any changes or
modifications are made.  At issue are the health and safety of employees and
community, as well as production costs from accidents/incidents due to Poor
SQA.  While some DOE Operation Offices and contractors impose strict SQA
controls, these controls are not of a uniform nor codified nature.  Therefore,
in the absence of specific SQA life-cycle control procedures promulgated by
HQ, it is strongly recommended that operations offices

     *    Review the relevant documents,
          - ANSI/ASME NQA-1
          - ASME NQA-2, Part 2.7
          - Draft DOE Order 1330
          - NUREG/CR-4640 and adopt the appropriate controls.

     *    Establish administrative controls which will ensure that
          changes, modifications and errors are properly tested and
          reported to all affected organizations.

     *    Be sure to receive, by agreement, Errata either from the
          manufacturer or responsible field organization as it
          becomes available.

     *    Perform independent verification calculations on all new
          software.



------------------------------------------------------------------------------
This publication is one of several series of bulletins published so that DOE
program managers and contractors can share information about potential
occupational safety problems relevant to DOE operations.  For more information
or additional copies, contact Eleanor Crampton, Performance Evaluation
Division, Office of Safety Compliance, Assistant Secretary for Environment,
Safety & Health, U.S. Department of Energy, Washington, DC 20545; telephone
FTS 233-3294, Commercial (301) 353-3294.
------------------------------------------------------------------------------
.
.
.