1296Air_Traffic_ControlA_Case_Study_in_Designing_for_High

129

Air Traffic Control

A Case Study in Designing for High Availability

The FAA has faced this problem [of complexity] throughout its decade- old attempt to replace the nation’s increasingly obsolete air traffic

control system. The replacement, called Advanced Automation System, combines all the challenges of computing in the 1990s. A program that

is more than a million lines in size is distributed across hundreds of computers and embedded into new and sophisticated hardware, all of

which must respond around the clock to unpredictable real-time events. Even a small glitch potentially threatens public safety.

— W. Wayt Gibbs [Gibbs 94]

Air traffic control (ATC) is among the most demanding of all software applica- tions. It is

hard real time

, meaning that timing deadlines must be met absolutely; it is

safety critical

, meaning that human lives may be lost if the system does not perform correctly; and it is

highly distributed

, requiring dozens of controllers to work cooperatively to guide aircraft through the airways system. In the United States, whose skies are filled with more commercial, private, and military aircraft than any other part of the world, ATC is an area of intense public scrutiny. Aside from the obvious safety issues, building and maintaining a safe, reliable airways system requires enormous expenditures of public money. ATC is a multibillion- dollar undertaking.

This chapter is a case study of one part of a once-planned, next-generation ATC system for the United States. We will see how its architecture—in particu- lar, a set of carefully chosen views (as in Chapter 2) coupled with the right tactics (as in Chapter 5)—held the key to achieving its demanding and wide-ranging requirements. Although this system was never put into operation because of bud- getary constraints, it was implemented and demonstrated that the system could meet its quality goals.

In the United States, air traffic is controlled by the Federal Aviation Admin- istration (FAA), a government agency responsible for aviation safety in general.

Bass.book Page 129 Thursday, March 20, 2003 7:21 PM

130

Part Two Creating an Architecture 6

—Air Traffic Control

The FAA is the customer for the system we will describe. As a flight progresses from its departure airport to its arrival airport, it deals with several ATC entities that guide it safely through each portion of the airways (and ground facilities) it is using.

Ground

control

coordinates the movement of aircraft on the ground at an air- port. Towers control aircraft flying within an airport’s

terminal control area

, a cylindrical section of airspace centered at an airport. Finally,

en route centers

divide the skies over the country into 22 large sections of responsibility. Consider an airline flight from Key West, Florida, to Washington, D.C.’s

Dulles Airport. The crew of the flight will communicate with Key West ground control to taxi from the gate to the end of the runway, Key West tower during takeoff and climb-out, and then Miami Center (the en route center whose airspace covers Key West) once it leaves the Key West terminal control area. From there the flight will be handed off to Jacksonville Center, Atlanta Center, and so forth, until it enters the airspace controlled by Washington Center. From Washington Center, it will be handed off to the Dulles tower, which will guide its approach and landing. When it leaves the runway, the flight will communicate with Dulles ground control for its taxi to the gate. This is an oversimplified view of ATC in the United States, but it suffices for our case study. Figure 6.1 shows the hand-off process, and Figure 6.2 shows the 22 en route centers.

The system we will study is called the Initial Sector Suite System (ISSS), which was intended to be an upgraded hardware and software system for the 22 en route centers in the United States. It was part of a much larger government procurement that would have, in stages, installed similar upgraded systems in the towers and ground control facilities, as well as the transoceanic ATC facilities.

FIGURE 6.1

Flying from point A to point B in the U.S. air traffic control system.

Courtesy of Ian Worpole/

Scientific American

, 1994.

Bass.book Page 130 Thursday, March 20, 2003 7:21 PM

131

The fact that ISSS was to be procured as only one of a set of strongly related systems had a profound effect on its architecture. In particular, there was great incentive to adopt common designs and elements where possible because the ISSS developer also intended to bid on the other systems. After all, these differ- ent systems (en route center, tower, ground control) share many elements: inter- faces to radio systems, interfaces to flight plan databases, interfaces to each other, interpreting radar data, requirements for reliability and performance, and so on. Thus, the ISSS design was influenced broadly by the requirements for all of the upgraded systems, not just the ISSS-specific ones. The complete set of upgraded systems was to be called the Advanced Automation System (AAS).

Ultimately, the AAS program was canceled in favor of a less ambitious, less costly, more staged upgrade plan. Nevertheless, ISSS is still an illuminating case study because, when the program was canceled, the design and most of the code were actually already completed. Furthermore, the architecture of the system (as well as most other aspects) was studied by an independent audit team and found to be well suited to its requirements. Finally, the system that was deployed instead of ISSS borrowed heavily from the ISSS architecture. For these reasons, we will present the ISSS architecture as an actual solution to an extremely diffi- cult problem.

FIGURE 6.2

En route centers in the United States

ZLC

ZDV ZOA

ZLA ZAB

ZKO

ZMP

ZAU

ZFW

ZHU

ZMA

ZME

ZOB

ZID

ZDC

ZTL

ZJX

ZNY ZBW

ZBW

ZSE

Introduction

Bass.book Page 131 Thursday, March 20, 2003 7:21 PM

132

Part Two Creating an Architecture 6

—Air Traffic Control

6.1 Relationship to the Architecture Business Cycle

Figure 6.3 shows how the air traffic control system relates to the Architecture Business Cycle (ABC). The end users are federal air traffic controllers; the cus- tomer is the Federal Aviation Administration; and the developing organization is a large corporation that supplies many other important software-intensive sys- tems to the U.S. government. Factors in the technical environment include the mandated use of Ada as the language of implementation for large government software systems and the emergence of distributed computing as a routine way to build systems and approach fault tolerance.

6.2 Requirements and Qualities

Given that air traffic control is highly visible, with huge amounts of commercial, government, and civilian interest, and given that it involves the potential loss of human life if it fails, its two most important quality requirements are as follows:

Ultrahigh availability, meaning that the system is absolutely prohibited from being inoperative for longer than very short periods. The actual availability

FIGURE 6.3

The ABC applied to the ATC system

Requirements (Qualities) High Availability High

Performance Openness Subsets Interoperability

Architect’s Influences

Stakeholders FAA and Air Traffic

Controllers

Developing Organization Government Contractors

Technical Environment Ada Distributed Computing

Architect’s Experience Fault Tolerant Message-Based Systems

Architect(s)

System ISSS

Architecture

Bass.book Page 132 Thursday, March 20, 2003 7:21 PM

Requirements and Qualities

133

requirement for ISSS is targeted at 0.99999, meaning that the system should be unavailable for less than 5 minutes a year. (However, if the system is able to recover from a failure and resume operating within 10 seconds, that fail- ure is not counted as unavailable time.)

High performance, meaning that the system has to be able to process large numbers of aircraft—as many as 2,440—without “losing” any of them. Net- works have to be able to carry the communication loads, and the software has to be able to perform its computations quickly and predictably.

In addition, the following requirements, although not as critical to the safety of the aircraft and their passengers, are major drivers in the shape of the architecture and the principles behind that shape:

�

Openness, meaning that the system has to be able to incorporate commer- cially developed software components, including ATC functions and basic computing services such as graphics display packages

�

The ability to field subsets of the system, to handle the case in which the billion-dollar project falls victim to reductions in budget (and hence func- tionality)—as indeed happened

�

The ability to make modifications to the functionality and handle upgrades in hardware and software (new processors, new I/O devices and drivers, new versions of the Ada compiler)

�

The ability to operate with and interface to a bewildering set of external systems, both hardware and software, some decades old, others not yet implemented

Finally, this system is unusual in that is must satisfy a great many stakeholders, particularly the controllers, who are the system’s end users. While this does not sound unusual, the difference is that controllers have the ability to reject the sys- tem if it is not to their liking, even if it meets all its operational requirements. The implications of this situation were profound for the processes of determining requirements and designing the system, and slowed it down substantially.

The term

sector suite

refers to a suite of controllers (each sitting at a control console like the one in Figure 6.4) that together control all of the aircraft in a par- ticular sector of the en route center’s airspace. Our oversimplified view of ATC is now enhanced by the fact that aircraft are handed off not only from center to cen- ter but also from sector to sector within each center. Sectors are defined in ways unique to each center. They may be defined to balance the load among the cen- ter’s controllers; for instance, less-traveled sectors may be larger than densely flown areas.

The ISSS design calls for flexibility in how many control stations are assigned to each sector; anywhere from one to four are allowed, and the number can be changed administratively while the system is in operation. Each sector is required to have at least two controllers assigned to it. The first is the radar con- troller, who monitors the radar surveillance data, communicates with the aircraft, and is responsible for maintaining safe separations. The controller is responsible for managing the tactical situation in the sector. The second controller is the data

Bass.book Page 133 Thursday, March 20, 2003 7:21 PM

134

Part Two Creating an Architecture 6

—Air Traffic Control

controller, who retrieves information (such as flight plans) about each aircraft that is either in the sector or soon will be. The data controller provides the radar controller with the information needed about the aircraft’s intentions in order to safely and efficiently guide it through the sector.

ISSS is a large system. Here are some numbers to convey a sense of scale:

�

ISSS is designed to support up to 210 consoles per en route center. Each console contains its own workstation-class processor; the CPU is an IBM RS/6000.

�

ISSS requirements call for a center to control from 400 to 2,440 aircraft tracks simultaneously.

�

There may be 16 to 40 radars to support a single facility.

�

A center may have from 60 to 90 control positions (each with one or several consoles devoted to it).

�

The code to implement ISSS contains about 1 million lines of Ada.

In summary, the ISSS system must do the following:

�

Acquire radar target reports that are stored in an existing ATC system called the Host Computer System.

�

Convert the radar reports for display and broadcast them to all of the con- soles. Each console chooses the reports that it needs to display; any console is capable of displaying any area.

FIGURE 6.4

Controllers at a sector suite.

Courtesy of William J. Hughes Technical Center; FAA public domain photo.

Bass.book Page 134 Thursday, March 20, 2003 7:21 PM

Architectural Solution

135

�

Handle conflict alerts (potential aircraft collisions) or other data transmitted by the host computer.

�

Interface to the Host for input and retrieval of flight plans.

�

Provide extensive monitoring and control information, such as network management, to allow site administrators to reconfigure the installation in real time.

�

Provide a recording capability for later playback.

�

Provide graphical user interface facilities, such as windowing, on the con- soles. Special safety-related provisions are necessary, such as window trans- parency to keep potentially crucial data from being obscured.

�

Provide reduced backup capability in the event of failure of the Host, the primary communications network, or the primary radar sensors.

In the next section, we will explore the architecture that fulfilled these requirements.

6.3 Architectural Solution

Just as an architecture affects behavior, performance, fault tolerance, and main- tainability, so it is shaped by stringent requirements in any of these areas. In the case of ISSS, by far the most important driving force is the extraordinarily high require- ment for system availability: less than 5 minutes per year of downtime. This requirement, more than any other, motivated architectural decisions for ISSS.

We begin our depiction of the ISSS architecture by describing the physical environment hosting the software. Then we give a number of software architec- ture views (as described in Chapter 2), highlighting the tactics (as described in Chapter 5) employed by each. During this discussion, we introduce a new view not previously discussed: fault tolerance. After discussing the relationships among views, we conclude the architecture picture for ISSS by introducing a refinement of the “abstract common services” tactic for modifiability and extensi- bility, namely, code templates.

ISSS PHYSICAL VIEW

ISSS is a distributed system, consisting of a number of elements connected by local area networks. Figure 6.5 shows a physical view of the ISSS system. It does not show any of the support systems or their interfaces to the ISSS equipment. Neither does it show any structure of the software. The major elements of the physical view and the roles its elements play are as follows:

�

The Host Computer System

is the heart of the en route automation system. At each en route center there are two host computers, one primary and the

Bass.book Page 135 Thursday, March 20, 2003 7:21 PM

136

Part Two Creating an Architecture 6

—Air Traffic Control

other ready to take over should there be some problem with the primary one. The Host provides processing of both surveillance and flight plan data. Sur- veillance data is displayed on the en route display consoles used by control- lers. Flight data is printed as necessary on flight strip printers, and some flight data elements are displayed on the data tags associated with the radar surveillance information.

�

Common consoles are the air traffic controller’s workstations. They provide displays of aircraft position information and associated data tags in a plan view format (the radar display), displays of flight plan data in the form of

FIGURE 6.5

ISSS physical view

HCS A

HCS B

EDARC

Dual LIU-Hs

ESI

3 LIU-Cs

Dual BCN M&C Consoles

Central Processor

3 LIU-Cs

Central Processor

3 LIU-Cs

Test and Training Subsystem

Bridge Bridge Bridge Bridge

Common Consoles (up to 210)

<<LCN>>

<<BCN>>

systems external to ISS

multiple common console access rings

Key: UML

Bass.book Page 136 Thursday, March 20, 2003 7:21 PM

Architectural Solution

137

electronic flight strips,

and a variety of other information displays. They also allow controllers to modify the flight data and to control the informa- tion being displayed and its format. Common consoles are grouped in sector suites of one to four consoles, with each sector suite serving the controller team for one airspace control sector.

�

The common consoles are connected to the Host computers by means of the Local Communications Network (LCN), the primary network of ISSS. Each Host is interfaced to the LCN via dual LCN interface units (each called LIU- H), which act as a fault-tolerant redundant pair.

�

The LCN is composed of four parallel token ring networks for redundancy and for balancing overall loading. One network supports the broadcast of surveillance data to all processors. One processor is used for point-to-point communications between pairs of processors; one provides a channel for dis- play data to be sent from the common consoles to recording units for layer playback; and one is a spare. Bridges provide connections between the net- works of the access rings and those of the backbone. The bridges also pro- vide the ability to substitute the spare ring for a failed ring and to make other alternative routings.

�

The Enhanced Direct Access Radar Channel (EDARC) provides a backup display of aircraft position and limited flight data block information to the en route display consoles. EDARC is used in the event of a loss of the dis- play data provided by the host. It provides essentially raw unprocessed radar data and interfaces to an ESI (External System Interface) processor.

�

The Backup Communications Network (BCN) is an Ethernet network using TCP/IP protocols. It is used for other system functions besides the EDARC interface and is also used as a backup network in some LCN failure conditions.

�

Both the LCN and the BCN have associated

Monitor-and-Control

(M&C) consoles. These give system maintenance personnel an overview of the state of the system and allow them to control its operation. M&C consoles are ordinary consoles that contain special software to support M&C functions and also provide the top-level or global availability management functions.

�

The Test and Training subsystem provides the capability to test new hardware and software and to train users without interfering with the ATC mission.

�

The central processors are mainframe-class processors that provide the data recording and playback functions for the system in an early version of ISSS.

A flight strip is a strip of paper, printed by the system that contains flight plan data about an aircraft currently in or about to arrive in a sector. Before ISSS, these flight strips were annotated by hand in pencil. ISSS was to provide the capability to manipulate strips onscreen.

Bass.book Page 137 Thursday, March 20, 2003 7:21 PM

138

Part Two Creating an Architecture 6

—Air Traffic Control

Each common console is connected to both the LCN and the BCN. Because of the large number of common consoles that may be present at a facility (up to 210), multiple LCN access rings are used to support all of them. This, then, is the physical view for ISSS, highlighting the hardware in which the software resides.

MODULE DECOMPOSITION VIEW

The module elements of the ISSS operational software are called Computer Soft- ware Configuration Items (CSCIs), defined in the government software development standard whose use was mandated by the customer. CSCIs correspond largely to work assignments; large teams are devoted to designing, building, and testing them. There is usually some coherent theme associated with each CSCI—some rationale for grouping all of the small software elements (such as packages, pro- cesses, etc.) that it contains.

There are five CSCIs in ISSS, as follows:

Display Management, responsible for producing and maintaining displays on the common consoles.

Common System Services, responsible for providing utilities generally use- ful in air traffic control software—recall that the developer was planning to build other systems under the larger AAS program.

Recording, Analysis, and Playback, responsible for capturing ATC sessions for later analysis.

National Airspace System Modification, entailing a modification of the soft- ware that resides on the Host (outside the scope of this chapter).

The IBM AIX operating system, providing the underlying operating system environment for the operational software.

These CSCIs form units of deliverable documentation and software, they appear in schedule milestones, and each is responsible for a logically related segment of ISSS functionality.

The module decomposition view reflects several modifiability tactics, as dis- cussed in Chapter 5. “Semantic coherence” is the overarching tactic for allocat- ing well-defined and nonoverlapping responsibilities to each CSCI. The Common System Services Module reflects the tactic of “abstract common services.” The Recording, Analysis, and Playback CSCI reflects the “record/playback” tactic for testability. The resources of each CSCI are made available through carefully designed software interfaces, reflecting “anticipation of expected changes,” “gen- eralizing the module,” and “maintaining interface stability

.”

PROCESS VIEW

The basis of concurrency in ISSS resides in elements called

applications

. An application corresponds roughly to a process, in the sense of Dijkstra’s cooperat- ing sequential processes, and is at the core of the approach the ISSS designers

Bass.book Page 138 Thursday, March 20, 2003 7:21 PM

Architectural Solution

139

adopted for fault tolerance. An application is implemented as an Ada “main” unit (a process schedulable by the operating system) and forms part of a CSCI (which helps us define a mapping between the module decomposition view and this one). Applications communicate by message passing, which is the connector in this component-and-connector view.

ISSS is constructed to operate on a plurality of processors. Processors (as described in the physical view) are logically combined to form a

processor group

, the purpose of which is to host separate copies of one or more applications. This concept is critical to fault tolerance and (therefore) availability. One executing copy is primary, and the others are secondary; hence, the different application copies are referred to as

primary address space

(PAS) or

standby address space

(SAS). The collection of one primary address space and its attendant standby address spaces is called an

operational unit

. A given operational unit resides entirely within the processors of a single processor group, which can consist of up to four processors. Those parts of the ISSS that are not constructed in this fault-tolerant manner (i.e., of coexisting primary and standby versions) simply run independently on different processors. These are called

functional groups

and they are present on each processor as needed, with each copy a separate instance of the program, maintaining its own state.

In summary, an application may be either an operating unit or a functional group. The two differ in whether the application’s functionality is backed up by one or more secondary copies, which keep up with the state and data of the pri- mary copy and wait to take over in case the primary copy fails. Operational units have this fault-tolerant design; functional groups do not. An application is imple- mented as an operational unit if its availability requirements dictate it; otherwise, it is implemented as a functional group.

Applications interact in a client-server fashion. The client of the transaction sends the server a

service request message

, and the server replies with an acknowl- edgment. (As in all client-server schemes, a particular participant—or application in this case—can be the client in one transaction and the server in another.) Within an operational unit, the PAS sends state change notifications to each of its SASs, which look for time-outs or other signs that they should take over and become primary if the PAS or its processor fails. Figure 6.6 summarizes how the primary and secondary address spaces of an application coordinate with each other to provide backup capability and give their relationship to processor groups.

When a functional group receives a message, it need only respond and update its own state as appropriate. Typically, the PAS of an operational unit receives and responds to messages on behalf of the entire operational unit. It then must update both its own state and the state of its SASs, which involves sending the SASs additional messages.

In the event of a PAS failure, a switchover occurs as follows:

A SAS is promoted to the new PAS.

The new PAS reconstitutes with the clients of that operational unit (a fixed list for each operational unit) by sending them a message that means,

Bass.book Page 139 Thursday, March 20, 2003 7:21 PM

140

Part Two Creating an Architecture 6

—Air Traffic Control

essentially: The operational unit that was serving you has had a failure. Were you waiting for anything from us at the time? It then proceeds to service any requests received in response.

A new SAS is started to replace the previous PAS.

The newly started SAS announces itself to the new PAS, which starts send- ing it messages as appropriate to keep it up to date.

If failure is detected within a SAS, a new one is started on some other processor. It coordinates with its PAS and starts receiving state data.

To add a new operational unit, the following step-by-step process is employed:

�

Identify the necessary input data and where it resides.

�

Identify which operational units require output data from the new opera- tional unit.

�

Fit this operational unit’s communication patterns into a systemwide acyclic graph in such a way that the graph remains acyclic so that deadlocks will not occur.

�

Design the messages to achieve the required data flows.

FIGURE 6.6

Functional groups (FG), operational units, processor groups, and primary/standby address spaces

Processor Group

Processor 3

<<FG>>

<<SAS>>

Processor 2

<<FG>>

<<PAS>>

Processor 1

<<FG>>

<<SAS>>

Operational Unit

Key: UML

Bass.book Page 140 Thursday, March 20, 2003 7:21 PM

Architectural Solution

141

■

Identify internal state data that must be used for checkpointing and the state data that must be included in the update communication from PAS to SAS.

■

Partition the state data into messages that fit well on the networks.

■

Define the necessary message types.

■

Plan for switchover in case of failure: Plan updates to ensure complete state.

■

Ensure consistent data in case of switchover.

■

Ensure that individual processing steps are completed in less time than a system “heartbeat.”

■

Plan data-sharing and data-locking protocols with other operational units.

This process is not for novices, but can be navigated straightforwardly by experi- enced team members. A tactic discussed in a section that follows—code tem- plates—was used to make the process more repeatable and much less error prone.

The process view reflects several availability tactics, including “state resyn- chronization,” “shadowing,” “active redundancy,” and “removal from service.”

CLIENT-SERVER VIEW

Because the applications in the process view interact with each other in client- server fashion, it is reasonable to show a client-server view of ISSS as well, although the behavior it describes largely mirrors that captured by the process view shown earlier. For completeness, Figure 6.7 shows a client-server view of the system.

FIGURE 6.7

Applications as clients and servers

<<SAS>><<SAS>><<PAS>>

Server Operational Unit

<<SAS>><<SAS>><<PAS>>

Client Operational Unit

Service Request Service

Request Response

Key: UML

Bass_ch06.fm Page 141 Monday, October 6, 2003 12:07 PM

142 Part Two Creating an Architecture 6—Air Traffic Control

The clients and servers were carefully designed to have consistent (as opposed to ad hoc) interfaces. This was facilitated by using simple message-passing protocols for interaction. The result reflects the modifiability tactics of “maintaining inter- face stability,” “component replacement,” and “adherence to defined protocols.”

CODE VIEW

One view not discussed in Chapter 2 but which sometimes appears in architec- tures of large systems is the code view. A code view shows how functionality is mapped to code units.

In ISSS, an Ada (main) program is created from one or more source files; it typically comprises a number of subprograms, some of which are gathered into separately compilable packages. The ISSS is composed of several such pro- grams, many of which operate in a client-server manner.

An Ada program may contain one or more tasks, which are Ada entities capable of executing concurrently with each other. These are the code-view corol- lary of the processes described in the process view. Because Ada tasks are managed by the Ada runtime system, ISSS also employs a mapping of Ada tasks onto UNIX (AIX) processes, which means that all individual threads of control (whether separate Ada programs or tasks within a single Ada program) are inde- pendent AIX processes operating concurrently.

Applications (i.e., operational units and functional groups) are decomposed into Ada packages, some of which include only type definitions and some of which are re-used across applications. Packaging is a design activity intended to embody abstraction and information hiding, and it is carried out by an opera- tional unit’s chief designer.

LAYERED VIEW

Underlying the operation of the ATC application programs on the ISSS proces- sors system is a commercial UNIX operating system, AIX. However, UNIX does not provide all the services necessary to support a fault-tolerant distributed sys- tem such as ISSS. Therefore, additional system services software was added. Fig- ure 6.8 shows as a set of layers the overall software environment in a typical ISSS processor.2

The lowest two rows of elements above AIX represent extensions to AIX that run within the AIX kernel’s address space. Because of performance requirements

2 Strictly speaking, Figure 6.8 is an overlay between a layered view and a component-and- connector view, because it shows runtime connections between the submodules in the lay- ers. In two cases, AAS Services and Other Device Driver, the connections among these and other submodules within the layered view are not shown, because there are so many that it would clutter the diagram. These services are freely used by most of the layered sys- tem. The actual connections would be listed in the supporting documentation for this view.

Bass.book Page 142 Thursday, March 20, 2003 7:21 PM

Architectural Solution 143

FIGURE 6.8 ISSS software architecture layers. The associations show data and/or control flow, making this an overlay of layers and a component-and- connector view.

Shared Memory (Tables and Message Storage)

Service Packages

Application Subroutines

Address Space Modules

Applications

LGSM Internal Time Switch

AAS Application

Prepare BCN Messages

SAR Data Collection

Prepare Messages

Atomic Broadcast Manager

OSI Stack

NISL

CAS AIX Kernel Extension

Other Device Driver

Ethernet Device Driver

Token Ring Device Driver

Shared Memory (Tables and Message Storage)

AIX Kernel

TCP/UDP IPStation ManagerAAS Services

Key: UML

Bass.book Page 143 Thursday, March 20, 2003 7:21 PM

144 Part Two Creating an Architecture 6—Air Traffic Control

and for compatibility with the AIX operating system, these extensions are gener- ally small programs written in the C language. Since they run within the kernels’ address space, faults in these programs can potentially damage AIX itself; hence, they must be relatively small, trusted programs reflecting the “limit exposure” tactic, discussed in Chapter 5. Although the tactic is security based—namely, to prevent denial of service—in ISSS it is used to enhance availability, which is a comple- mentary goal. Happily, sometimes tactics serve multiple quality attributes well.

The Atomic Broadcast Manager (ABM) plays a key role in the communica- tion among the Local Availability Manager modules within a sector suite to man- age the availability of suite functions. The Station Manager provides datagram services on the LCN and serves as the local representative of the LCN network management services. The Network Interface Sublayer provides a similar func- tion for the point-to-point messages, sharing its network information with the Station Manager.

The next two layers represent operating system extensions that execute out- side the AIX kernel’s address space and therefore cannot directly damage AIX if they contain faults. These programs are generally written in Ada.

Prepare Messages handles LCN messages for application programs. Prepare BCN Messages performs a similar function for messages to be sent on the BCN. One function of these programs is to determine which of the multiple redundant copies of an application program within a sector suite is the primary and thus is to receive messages. The Local Availability Manager provides the control infor- mation needed to make this determination.

The top layer is where the applications reside. The Local Availability Man- ager and the Internal Time Synchronization programs are application-level system services. The Local Availability Manager is responsible for managing the initia- tion, termination, and availability of the application programs. It communicates with each address space on its own processor to control its operation and check its status. It also communicates with the Local Availability Manager on the other processors within its sector suite to manage the availability of suite functions, including switching from a primary to a backup copy of an application program when appropriate. The Local Availability Manager communicates with the Global Availability Management application that resides on the M&C consoles to report status and to accept control commands. The Internal Time Synchronization pro- gram synchronizes the processor’s clock with that of the other ISSS processors, which is crucial to the operation of the availability management functions. (See the fault tolerance view, in Figure 6.9.)

A NEW VIEW: FAULT TOLERANCE

As we said, the views listed in Chapter 2 are not exhaustive. In fact, there is no exhaustive list of views that constitute the complete software architecture for all systems or for any system. A welcome trend in software architecture is the recog- nition of the importance of architecture in achieving quality attributes, and there-

Bass.book Page 144 Thursday, March 20, 2003 7:21 PM

Architectural Solution 145

fore the importance of explicitly stating the quality attributes that the architecture is to provide. Toward this end, architects often produce views that show how the architecture achieves a particular quality attribute: a security view, for example. For runtime qualities, these views are in the component-and-connector category, showing runtime element interactions. For non-runtime qualities, these views are in the module category, showing how the implementation units are designed to achieve (for example) modifiability.

The high availability requirements for ISSS elevated fault tolerance to an important role in the design of the system. For one thing, a cold system restart in

FIGURE 6.9 ISSS component-and-connector view for fault tolerance

Key: UML

Global Availability Manager

Exceptions

Local/Group Availability Manager

Network

M&C Console

Network Manager Performance Monitor

ATC Console

Application Software Operational Unit Thread Processing Model

O/S Extensions Address Space Models Runtime Environment

Operating System

Processor I/O Devices

Attachments

Exceptions

ExceptionsExceptions

Exceptions

Bass.book Page 145 Thursday, March 20, 2003 7:21 PM

146 Part Two Creating an Architecture 6—Air Traffic Control

the event of a failure was out of the question. Immediate (or at least rapid) switchover to a component on standby seemed the best approach. As design pro- gressed and this idea became clearer, a new architectural structure emerged: the fault-tolerant hierarchy (Figure 6.9). This structure describes how faults are detected and isolated and how the system recovers. Whereas the PAS/SAS scheme traps and recovers from errors that are confined within a single applica- tion, the fault-tolerant hierarchy is designed to trap and recover from errors that are the result of cross-application interaction.

The ISSS fault-tolerant hierarchy provides various levels of fault detection and recovery. Each level asynchronously

� Detects errors in self, peers, and lower levels. � Handles exceptions from lower levels. � Diagnoses, recovers, reports, or raises exceptions.

Each level is meant to produce another increment in system availability above that produced by the lower levels. The levels are as follows:

� Physical (network, processor, and I/O devices) � Operating system � Runtime environment � Application � Local availability � Group availability � Global availability � System monitor and control

Fault detection and isolation are performed at each level in the hierarchy. Fault detection is by built-in tests, event time-outs, network circuit tests, group mem- bership protocol, and, as a last resort, human reaction to alarms and indicators.

Fault recovery is performed at each level in the software hierarchy and can be automatic or manual. For the Local, Group, and Global Availability managers, the recovery methods are table driven. In a PAS, there are four types of recovery from failure. The type of recovery used depends on the current operational status and is determined by the Local Availability Manager using decision tables, as follows:

� In a switchover, the SAS takes over almost immediately from its PAS. � A warm restart uses checkpoint data (written to nonvolatile memory). � A cold restart uses default data and loses state history. � A cutover is used to transition to new (or old) logic or adaptation data.

Redundancy is provided by network hardware (LCN, BCN, and associated bridges), processor hardware (up to four processors per processor group, redun- dant recording), and software (multiple address spaces per operational unit).

In addition to the availability tactics already seen with the process view, the fault tolerance view adds “ping/echo” and “heartbeat” as ways to detect failures,

Bass.book Page 146 Thursday, March 20, 2003 7:21 PM

Architectural Solution 147

exception to percolate errors to the appropriate place for correction, and spare to perform recovery.

RELATING THE VIEWS TO EACH OTHER

During the preceding discussion, the elements in one view made “guest appear- ances” in other views. Although views form the backbone of understanding a sys- tem, deeper insight is often gained by examining the relations the views have to each other and, in particular, from examining mappings from view to view. This imparts a more holistic view of the architecture.

In ISSS, CSCIs are elements in the module decomposition view. They are composed of applications, which in turn are elements in the process view and the client-server view. Applications are implemented as Ada programs and packages, shown in the code view, which in turn map to threads, which are elements in the concurrency view (not shown). The layered view describes the functionality assigned to the modules in the decomposition view in a way that shows what they are allowed to use. Finally, a specialized view focusing on the achievement of a particular runtime quality attribute—the fault tolerance view—uses the elements of the process, layer, and module views.

Chapter 9, which covers how to document a software architecture, will pre- scribe a special place in the documentation package for capturing view relation- ships. For ISSS, that mapping would include tables that list the elements from the various views and show how they correspond to each other as described above.

ADAPTATION DATA

ISSS makes extensive use of the modifiability tactic of “configuration files,” which it calls adaptation data. Site-specific adaptation data tailors the ISSS sys- tem across the 22 en route centers in which it was planned to be deployed, and so-called preset adaptation data tailors the software to changes that arise during development and deployment but which do not represent site-specific differences. Adaptation data represents an elegant and crucial shortcut to modifying the sys- tem in the face of site-specific requirements, user- or center-specific preferences, configuration changes, requirements changes, and other aspects of the software that might be expected to vary over time and across deployment sites. In effect, the software has been designed to read its operating parameters and behavioral spec- ifications from input data; it is therefore completely general with respect to the set of behaviors that can be represented in that data (reflecting the “generalize the module” tactic). For example, a requirements change to split the data in one ATC window view into two separate windows—a nontrivial change in many systems— could be accomplished by changing the adaptation data and a few lines of code.

The negative side is that adaptation data presents a complicated mechanism to maintainers. For example, although it is trivial (from an operational point of

Bass.book Page 147 Thursday, March 20, 2003 7:21 PM

148 Part Two Creating an Architecture 6—Air Traffic Control

view) to add new commands or command syntax to the system, the implementa- tion of this flexibility is in fact a complicated interpretive language all its own. Also, complicated interactions may occur between various pieces of adaptation data, which could affect correctness, and there are no automated or semiauto- mated mechanisms in place to guard against the effects of such inconsistencies. Finally, adaptation data significantly increases the state space within which the operational software must correctly perform, and this has broad implications for system testing.

REFINING THE “ABSTRACT COMMON SERVICES” TACTIC: CODE TEMPLATES FOR APPLICATIONS

Recall that the primary–secondary address space scheme described earlier relies on redundancy to achieve fault tolerance: Copies of the software are stored on different processors. While the primary copy is executing, it sends state informa- tion from time to time to all of the secondary copies so that they can take up exe- cution when called on. The implementation plan for these copies called for both to come from true copies of the same source code. Even though the primary and secondary copies are never doing the same thing at the same time (the primary is performing its duty and sending state updates to its backups, and the secondaries are waiting to leap into action and accepting state updates), both programs come from identical copies of the same source code. To accomplish this, the contractor developed a standard code template for each application; the template is illus- trated in Figure 6.10.

The structure is a continuous loop that services incoming events. If the event is one that causes the application to take a normal (non-fault-tolerant-related) action, it carries out the appropriate action, followed by an update of its backup counterparts’ data so that the counterpart can take over if necessary. Most appli- cations process between 50 and 100 normal events. Other events involve the transfer (transmission and reception) of state and data updates. The last set of events involves both the announcement that this unit has become the primary address space and requests from clients for services that the former (now failed) primary address space did not complete.

This template has architectural implications: It makes it simple to add new applications to the system with a minimum of concern for the actual workings of the fault-tolerant mechanisms designed into the approach. Coders and maintain- ers of applications do not need to know about message-handling mechanisms except abstractly, and they do not need to ensure that their applications are fault tolerant—that has been handled at a higher (architectural) level of design.

Code templates represent a refinement of the “abstract common services” tactic; the part of each application that is common is instantiated in the template. This tactic is related to several other tactics for modifiability. It reflects an “antici- pation of expected changes” in the parts it leaves variable and it gives the processes

Bass.book Page 148 Thursday, March 20, 2003 7:21 PM

Architectural Solution 149

a “semantic coherence,” because they all do the same thing when viewed abstractly. The template lets programmers concentrate on the details of their application, leading to “generalizing the module.” And by making the interfaces and protocols part of the template, they “maintain interface stability” and achieve “adherence to defined protocols.”

Table 6.1 summarizes the approaches and tactics by which the ISSS soft- ware architecture met its quality goals.

FIGURE 6.10 Code structure template for fault-tolerant ISSS applications

terminate:= false

initialize application/application protocols

ask for current state (image request)

Loop

Get_event

Case Event_Type is

-- "normal" (non-fault-tolerant-related) requests to perform actions;

-- only happens if this unit is the current primary address space

when X=> Process X

Send state data updates to other address spaces

when Y=>Process Y

Send state data updates to other address spaces

...

when Terminate_Directive => clean up resources; terminate := true

when State_Data_Update => apply to state data

-- will only happen if this unit is a secondary address space, receiving

-- the update from the primary after it has completed a "normal" action

-- sending, receiving state data

when Image_Request => send current state data to new address space

when State_Data_Image => Initialize state data

when Switch_Directive => notify service packages of change in rank

-- these are requests that come in after a PAS/SAS switchover; they

-- report services that they had requested from the old (failed) PAS

-- which this unit (now the PAS) must complete. A,B, etc. are the names

-- of the clients.

when Recon_from_A=>reconstitute A

when Recon_from_B=>reconstitute B

...

when others=>log error

end case

exit when terminate

end loop

Bass.book Page 149 Thursday, March 20, 2003 7:21 PM

150 Part Two Creating an Architecture 6—Air Traffic Control

6.4 Summary

Like all of the case studies in this book, ISSS illustrates how architectural solu- tions can be the key to achieving the needs of an application. Table 6.1 summa- rized the key approaches used. Because of its projected long life, high cost, large size, important function, and high visibility, ISSS was subject to extraordinary change pressures over and above its demanding operational requirements. Human– computer interfaces, new hardware, new commercial components, operating system and network upgrades, and capacity increases were not just likely but foregone conclusions. The architecture, by using a wide range of fault tolerance mechanisms (and code templates), including hardware and software redundancy and layered fault detection, and by using distributed multiprocess computing with client- server message passing, was able to satisfy its complex, wide-ranging operational requirements.

A footnote to our story is the intensive software audit that was carried out on the ISSS architecture by a blue-ribbon panel of experts when the U.S. govern- ment was considering abandoning ISSS in favor of a simpler, less expensive solu- tion. The audit assessed the architecture’s ability to deliver the required performance and availability and included modifiability exercises that walked through several change scenarios, including the following:

TABLE 6.1 How the ATC System Achieves Its Quality Goals

Goal How Achieved Tactic(s) Used

High Availability Hardware redundancy (both processor and network); soft- ware redundancy (layered fault detection and recovery)

State resynchronization; shadow- ing; active redundancy; removal from service; limit exposure; ping/ echo; heartbeat; exception; spare

High Performance Distributed multiprocessors; front-end schedulability analy- sis, and network modeling

Introduce concurrency

Openness Interface wrapping and layering

Abstract common services; main- tain interface stability

Modifiability Templates and table-driven adaptation data; careful assignment of module responsbilities; strict use of specified interfaces

Abstract common services; semantic coherence; maintain interface stability; anticipate expected changes; generalize the module; component replacement; adherence to defined procotols; configuration files

Ability to Field Subsets

Appropriate separation of concerns

Abstract common services

Interoperability Client-server division of func- tionality and message-based communications

Adherence to defined protocols; maintain interface stability

Bass.book Page 150 Thursday, March 20, 2003 7:21 PM

Discussion Questions 151

� Making major modifications to the M&C position’s human–computer interface

� Importing third-party-developed air traffic applications into the ISSS system � Adding new ATC views to the system � Replacing the RS/6000 processors with a chip of similar capability � Deleting electronic flight strips from the requirements � Increasing the system’s maximum capacity of flight tracks by 50 percent

In every case, the audit found that the ISSS software architecture had been designed so that the modifications would be straightforward and, in some cases, almost trivial. This is a tribute to its careful design and its explicit consideration of quality attributes and the architectural tactics to achieve them.

6.5 For Further Reading

The saga of the FAA’s attempts to upgrade its air traffic control software has been written about extensively; for example, by [Gibbs 94]. The effort to audit the ISSS system for salvageability was reported by [Brown 95]. In these papers, maintainability is treated as a dual quality related not only to the properties of the system but also to the capabilities of the organization slated to perform the main- tenance. This important aspect of maintainability—the necessary fit between the maintenance that a system needs and the maintenance that an organization is pre- pared to provide for it—is not usually discussed.

6.6 Discussion Questions

1. High availability was a main impetus behind the architecture presented in this chapter. How were other quality attributes, such as performance, affected by this requirement? How might the architecture change if this requirement were removed?

2. How many architectural patterns can you recognize in the architecture for ISSS?

3. Construct quality attribute scenarios, as described in Chapter 4, for as many of the requirements given in Section 6.2 as you can. Where necessary infor- mation is missing, propose reasonable substitutions.

Bass.book Page 151 Thursday, March 20, 2003 7:21 PM

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

Scenario-Based Usability Engineering

Mary Beth Rosson and John M. Carroll Department of Computer Science

Virginia Tech

Fall 1999

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 1

Chapter 3

Analyzing Requirements

Making work visible. The end goal of requirements analysis can be elusive when work is not understood in the same way by all participants. Blomberg, Suchman, and Trigg describe this problem in their exploration of image-processing services for a law firm. Initial studies of attorneys produced a rich analysis of their document processing needs—for any legal proceeding, documents often numbering in the thousands are identified as “responsive” (relevant to the case) by junior attorneys, in order to be submitted for review by the opposing side. Each page of these documents is given a unique number for subsequent retrieval. An online retrieval index is created by litigation support workers; the index encodes document attributes such as date, sender, recipient, and type. The attorneys assumed that their job (making the subjective relevance decisions) would be facilitated by image processing that encodes a documents’s objective attributes (e.g., date, sender). However, studies of actual document processing revealed activities that were not objective at all, but rather relied on the informed judgment of the support staff. Something as simple as a document date was often ambiguous, because it might display the date it was written, signed, and/or delivered; the date encoded required understanding the document’s content and role in a case. Even determining what constituted a document required judgment, as papers came with attachments and no indication of beginning or end. Taking the perspective of the support staff revealed knowledge-based activities that were invisible to the attorneys, but that had critical limiting implications for the role of image-processing technologies (see Blomberg, 1995).

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 2

What is Requirements Analysis?

The purpose of requirements analysis is to expose the needs of the current situation with respect to a proposed system or technology. The analysis begins with a mission statement or orienting goals, and produces a rich description of current activities that will motivate and guide subsequent development. In the legal office case described above, the orienting mission was possible applications of image processing technology; the rich description included a view of case processing from both the lawyers’ and the support staffs’ perspectives. Usability engineers contribute to this process by analyzing what and how features of workers’ tasks and their work situation are contributing to problems or successes1. This analysis of the difficulties or opportunities forms a central piece of the requirements for the system under development: at the minimum, a project team expects to enhance existing work practices. Other requirements may arise from issues unrelated to use, for example hardware cost, development schedule, or marketing strategies. However these pragmatic issues are beyond the scope of this textbook. Our focus is on analyzing the requirements of an existing work setting and of the workers who populate it.

Understanding Work

What is work? If you were to query a banker about her work, you would probably get a list of things she does on a typical day, perhaps a description of relevant information or tools, and maybe a summary of other individuals she answers to or makes requests of. At the least, describing work means describing the activities, artifacts (data, documents, tools), and social context (organization, roles, dependencies) of a workplace. No single observation or interview technique will be sufficient to develop a complete analysis; different methods will be useful for different purposes.

Tradeoff 3.1: Analyzing tasks into hierarchies of sub-tasks and decision rules brings order to a problem domain, BUT tasks are meaningful only in light of organizational goals and activities.

A popular approach to analyzing the complex activities that comprise work is to enumerate and organize tasks and subtasks within a hierarchy (Johnson, 1995). A banker might indicate that the task of “reviewing my accounts” consists of the subtasks “looking over the account list”, “noting accounts with recent activity”, and “opening and reviewing active accounts”. Each of these sub-tasks in turn can decomposed more finely, perhaps to the level of individual actions such as picking up or filing a particular document. Some of the tasks will include decision-making, such

1 In this discussion we use “work” to refer broadly to the goal-directed activities that take place in the

problem domain. In some cases, this may involve leisure or educational activities, but in general the same methods

can be applied to any situation with established practices.

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 3

as when the banker decides whether or not to open up a specific account based on its level of activity.

A strength of task analysis is its step-by-step transformation of a complex space of activities into an organized set of choices and actions. This allows a requirements analyst to examine the task’s structure for completeness, complexity, inconsistencies, and so on. However the goal of systematic decomposition can also be problematic, if analysts become consumed by representing task elements, step sequences, and decision rules. Individual tasks must be understood within the larger context of work; over-emphasizing the steps of a task can cause analysts to miss the forest for the trees. To truly understand the task of reviewing accounts a usability engineer must learn who is responsible for ensuring that accounts are up to date, how account access is authorized and managed, and so on.

The context of work includes the physical, organizational, social, and cultural relationships that make up the work environment. Actions in a workplace do not take place in a vacuum; individual tasks are motivated by goals, which in turn are part of larger activities motivated by the organizations and cultures in which the work takes place (see Activities of a Health Care Center, below). A banker may report that she is reviewing accounts, but from the perspective of the banking organization she is “providing customer service” or perhaps “increasing return on investment”. Many individuals — secretaries, data-entry personnel, database programmers, executives — work with the banker to achieve these high-level objectives. They collaborate though interactions with shared tools and information; this collaboration is shaped not only by the tools that they use, but also by the participants’ shared understanding of the bank’s business practice — its goals, policies, and procedures.

Tradeoff 3.2: Task information and procedures are externalized in artifacts, BUT the impact of these artifacts on work is apparent only in studying their use.

A valuable source of information about work practices is the artifacts used to support task goals (Carroll & Campbell, 1989). An artifact is simply a designed object — in an office setting, it might be a paper form, a pencil, an in-basket, or a piece of computer software. It is simple and fun to collect artifacts and analyze their characteristics (Norman, 1990). Consider the shape of a pencil: it conveys a great deal about the size and grasping features of the humans who use it; pencil designers will succeed to a great extent by giving their new designs the physical characteristics of pencils that have been used for years. But artifacts are just part of the picture. Even an object as simple as a pencil must be analyzed as part of a real world activity, an activity that may introduce concerns such as erasability (elementary school use), sharpness (architecture firm drawings), name-brands (pre-teen status brokering), cost (office supplies accounting), and so on.

Usability engineers have adapted ethnographic techniques to analyze the diverse factors influencing work. Ethnography refers to methods developed within anthropology for gaining insights into the life experiences of individuals whose everyday reality is vastly different from the

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 4

analyst’s (Blomberg, 1990). Ethnographers typically become intensely involved in their study of a group’s culture and activities, often to the point of becoming members themselves. As used by HCI and system design communities, ethnography involves observations and interviews of work groups in their natural setting, as well as collection and analysis of work artifacts (see Team Work in Air Traffic Control, below). These studies are often carried out in an iterative fashion, where the interpretation of one set of data raises questions or possibilities that may be pursued more directly in follow-up observations and interviews.

Figure 3.1: Activity Theory Analysis of a Health Care Center (after Kuuiti and Arvonen, 1992)

Activities of a Health Care Center: Activity Theory (AT) offers a view of individual work that grounds it in the goals and practices of the community within which the work takes place. Engeström (1987) describes how an individual (the subject) works on a problem (the object) to achieve a result (the outcome), but that the work on the problem is mediated by the tools available (see Figure 3.2m). An individual’s work is also mediated by the rules of practice shared within her community; the object of her work is mediated by that same communities division of labor.

Kuutti and Arvonen (1992; see also Engeström 1990; 1991; 1993) applied this framework to their studies of a health care organization in Espoo, Finland. This organization wished to evolve

Tools Supporting Activity:

Subject Involved in Activity:

Community sponsoring Activity:

Object of Activity:

Activity Outcome:

Division of LaborRules of Practice

patient record, medicines, etc.

one physician in a health care unit

all personnel of the health care unit

the complex, multi-dimensional problem of a patient

patient problem resolved

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 5

from a rather bureaucratic organization with strong separations between its various units (e.g., social work, clinics, hospital) to a more service-oriented organization. A key assumption in doing this was that the different units shared a common general object of work—the “life processes” of the town’s citizens. This high-level goal was acknowledged to be a complex problem requiring the integrated services of complementary health care units.

The diagram in Figure 3.1 summarizes an AT analysis developed for one physician in a clinic. The analysis records the shared object (the health conditions of a patient). At the same time it shows this physician’s membership in a subcommunity, specifically the personnel at her clinic. This clinic is both geographically and functionally separated from other health care units, such as the hospital or the social work office. The tools that the physician uses in her work, the rules that govern her actions, and her understanding of her goals are mediated by her clinic. As a result, she has no way of analyzing or finding out about other dimensions of this patient’s problems, for example the home life problems being followed by a social worker, or emotional problems under treatment by psychiatric personnel. In AT such obstacles are identified as contradictions which must be resolved before the activity can be successful.

In this case, a new view of community was developed for the activity. For each patient, email or telephone was used to instantiate a new community, comprised of individuals as relevant from different health units. Of course the creation of a more differentiated community required negotiation concerning the division of labor (e.g. who will contact whom and for what purpose), and rules of action (e.g., what should be done and in what order). Finally, new tools (composite records, a “master plan”) were constructed that better supported the redefined activity.

Figure 3.2 will appear here, a copy of the figure provided by Hughes et al. in their ethnographic report. Need to get copyright permission.

Team Work in Air Traffic Control: An ethnographic study of British air traffic control rooms by Hughes, Randall and Shapiro (CSCW’92) highlighted the central role played by the paper strips used to chart the progress of individual flights. In this study the field workers immersed themselves in the work of air traffic controllers for several months. During this time they observed the activity in the control rooms and talked to the staff; they also discussed with the staff the observations they were collecting and their interpretation of these data.

The general goal of the ethnography was to analyze the social organization of the work in the air traffic control rooms. In this the researchers showed how the flight progress strips supported “individuation”, such that each controller knew what their job was in any given situation, but also how their tasks were interdependent with the tasks of others. The resulting division of labor was accomplished in a smooth fashion because the controllers had shared knowledge of what the strips indicated, and were able to take on and hand off tasks as needed, and to recognize and address problems that arose.

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 6

Each strip displays an airplane’s ID and aircraft type; its current level, heading, and airspeed; its planned flight path, navigation points on route, estimated arrival at these points; and departure and destination airports (see Figure 3.2). However a strip is more than an information display. The strips are work sites, used to initiate and perform control tasks. Strips are printed from the online database, but then annotated as flight events transpire. This creates a public history; any controller can use a strip to reconstruct a “trajectory” of what the team has done with a flight. The strips are used in conjunction with the overview offered by radar to spot exceptions or problems to standard ordering and arrangement of traffic. An individual strip gets “messy” to the extent it has deviated from the norm, so a set of strips serves as a sort of proxy for the orderliness of the skies.

The team interacts through the strips. Once a strip is printed and its initial data verified, it is placed in a holder color-coded for its direction. It may then be marked up by different controllers, each using a different ink color; problems or deviations are signaled by moving a strip out of alignment, so that visual scanning detects problem flights. This has important social consequences for the active controller responsible for a flight. She knows that other team members are aware of the flight’s situation and can be consulted; who if anyone has noted specific issues with the flight; if a particularly difficult problem arises it can be passed on to the team leader without a lot of explanation; and so on.

The ethnographic analysis documented the complex tasks that revolved around the flight control strips. At the same time it made clear the constraints of these manually-created and maintained records. However a particularly compelling element of the situation was the controllers’ trust in the information on the strips. This was due not to the strips’ physical characteristics, but rather to the social process they enable—the strips are public, and staying on top of each others’ problem flights, discussing them informally while working or during breaks, is taken for granted. Any computerized replacement of the strips must support not just management of flight information, but also the social fabric of the work that engenders confidence in the information displayed.

User Involvement

Who are a system’s target users? Clearly this is a critical question for a user-centered development process. It first comes up during requirements analysis, when the team is seeking to identify a target population(s), so as to focus in on the activities that will suggest problems and concerns. Managers or corporation executives are a good source of high-level needs statements (e.g., reduce data-processing errors, integrate billing and accounting). Such individuals also have a well-organized view of their subordinates’ responsibilities , and of the conditions under which various tasks are completed. Because of the hierarchical nature of most organizations, such individuals are usually easily to identify and comprise a relatively small set. Unfortunately if a requirements team accepts these requirements too readily, they may miss the more detailed and situation-specific needs of the individuals who will use a new system in their daily work.

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 7

Tradeoff 3.3: Management understands the high-level requirements for a system, BUT is often unaware of workers’ detailed needs and preferences.

Every system development situation includes multiple stakeholders (Checklund, 1981). Individuals in management positions may have authorized a system’s purchase or development; workers with a range of job responsibilities will actually use the system; others may benefit only indirectly from the tasks a system supports. Each set of stakeholders has its own set of motivations and problems that the new system might address (e.g., productivity, satisfaction, ease of learning). What’s more, none of them can adequately communicate the perspectives of the others — despite the best of intentions, many details of a subordinate’s work activities and concerns are invisible to those in supervisory roles. Clearly what is needed in requirements analysis is a broad-based approach that incorporates diverse stakeholder groups into the observation and interviewing activities.

Tradeoff 3.4: Workers can describe their tasks, BUT work is full of exceptions, and the knowledge for managing exceptions is often tacit and difficult to externalize.

But do users really understand their own work? We made the point above that a narrow focus on the steps of a task might cause analysts to miss important workplace context factors. An analogous point holds with respect to interviews or discussions with users. Humans are remarkably good (and reliable) at “rationalizing” their behaivor (Ericsson & Simon, 1992). Reports of work practices are no exception — when asked workers will usually first describe a most-likely version of a task. If an established “procedures manual” or other policy document exists, the activities described by experienced workers will mirror the official procedures and policies. However this officially-blessed knowledge is only part of the picture. An experienced worker will also have considerable “unofficial” knowledge acquired through years of encountering and dealing with the specific needs of different situations, with exceptions, with particular individuals who are part of the process, and so on. This expertise is often tacit, in that the knowledgeable individuals often don’t even realize what they “know” until confronted with their own behavior or interviewed with situation-specific probes (see Tacit Knowledge in Telephone Trouble-Shooting, below). From the perspective of requirements analysis, however, tacit knowledge about work can be critical, as it often contains the “fixes” or “enhancements” that have developed informally to address the problems or opportunities of day-to-day work.

One effective technique for probing workers’ conscious and unconscious knowledge is contextual inquiry (Beyers & Holtzblatt, 1994). This analysis method is similar to ethnography, in that it involves the observation of individuals in the context of their normal work environment. However it includes the perogative to interrupt an observed activity at points that seem informative (e.g., when a problematic situation arises) and to interview the affected individual(s) on the spot concerning the events that have been observed, to better understand causal factors and options for continuing the activity. For example, a usability engineer who saw a secretary stop working on a

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 8

memo to make a phone call to another secretary, might ask her afterwards to explain what had just happened between her and her co-worker.

Tacit Knowledge in Telephone Trouble-Shooting: It is common for workers to see their conversations and interactions with each other as a social aspect of work that is enjoyable but unrelated to work goals. Sachs (199x) observed this in her case study of telephony workers in a phone company. The study analyzed the work processes related to detecting, submitting, and resolving problems on telephone lines; the focus of the study was the Trouble Ticketing System (TTS), a large database used to record telephone line problems, assign problems (tickets) to engineers for correction, and keep records of problems detected and resolved.

Sachs argues that TTS takes an organizational view of work, treating work tasks as modular and well-defined: one worker finds a problem, submits it to the database, TTS assigns it to the engineer at the relevant site, that engineer picks up the ticket, fixes the problem, and moves on. The original worker is freed from the problem analysis task once the original ticket, and the second worker can move on once the problem has been addressed. TTS replaced a manual system in which workers contacted each other directly over the phone, often working together to resolve a problem. TTS was designed to make work more efficient by eliminating unnecessary phone conversations.

In her interviews with telephony veterans, Sachs discovered that the phone conversations were far from unnecessary. The initiation, conduct, and consequences of these conversations reflected a wealth of tacit knowledge on the part of the worker--selecting the right person to call (one known to have relevant expertise for this apparent problem), the “filling in” on what the first worker had or had not determined or tried to this point, sharing of hypotheses and testing methods, iterating together through tests and results, and carrying the results of this informal analysis into other possibly related problem areas. In fact, TTS had made work less efficient in many cases, because in order to do a competent job, engineers developed “workarounds” wherein they used phone conversations as they had in the past, then used TTS to document the process afterwards.

Of interest was that the telephony workers were not at first aware of how much knowledge of trouble-shooting they were applying to their jobs. They described the tasks as they understood them from company policy and procedures. Only after considerable data collection and discussion did they recognize that their jobs included the skills to navigate and draw upon a rich organizational network of colleagues. In further work Sachs helped the phone company to develop a fix for the observed workarounds in the form of a new organizational role: a “turf coordinator”, a senior engineer responsible for identifying and coordinating the temporary network of workers needed to collaborate on trouble-shooting a problem. As a result of Sach’s analysis, work that had been tacit and informal was elevated to an explicit business responsibility.

Requirements Analysis with Scenarios

As introduced in Chapter 2, requirements refers to the first phase of SBUE. As we also have emphasized, requirements cannot be analyzed all at once in waterfall fashion. However some

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 9

analysis must happen early on to get the ball rolling. User interaction scenarios play an important role in these early analysis activities. When analysts are observing workers in the world, they are collecting observed scenarios, episodes of actual interaction among workers that may or may not involve technology. The analysis goal is to produce a summary that captures the critical aspects of the observed activities. A central piece of this summary analysis is a set of requirements scenarios.

The development of requirements scenarios begins with determining who are the stakeholders in a work situation — what their roles and motivations are, what characteristics they possess that might influence reactions to new technology. A description of these stakeholders’ work practice is then created, through a combination of workplace observation and generation of hypothetical situations. These sources of data are summarized and combined to generate the requirements scenarios. A final step is to call out the most critical features of the scenarios, along with hypotheses about the positive or negative consequences that these features seem to be having on the work setting.

Introducing the Virtual Science Fair Example Case

The methods of SBUE will be introduced with reference to a single open-ended example problem, the design of a virtual science fair (VSF). The high-level concept is to use computer- mediated communication technology (e.g., email, online chat, discussion forums, videoconferencing) and online archives (e.g., databases, digital libraries) to supplement the traditional physical science fairs. Such fairs typically involve student creation of science projects over a period of months. The projects are then exhibited and judged at the science fair event. We begin with a very loose concept of what a virtual version of such a fair might be — not a replacement of current fairs, but rather a supplement that expands the boundaries of what might constitute participation, project construction, project exhibits, judging, and so on.

Stakeholder Analysis

Checklund (1981) offers a mnemonic for guiding development of an early shared vision of a system’s goals — CATWOE analysis. CATWOE elements include Clients (those people who will benefit or suffer from the system), Actors (those who interact with the system), a Transformation (the basic purpose of the system), a Weltanschauung (the world view promoted by the system), Owners (the individuals commissioning or authorizing the system), and the Environment (physical constraints on the system). SBUE adapts Checklund’s technique as an aid in identifying and organizing the concerns of various stakeholders during requirements analysis.The SBUE adaptation of Checklund’s technique includes the development of thumbnail scenarios for each element identified. The table includes just one example for each VSF element called out in the analysis; for a complex situation multiple thumbnails might be needed. Each scenario sketch is a usage-oriented elaboration of the element itself; the sketch is points to a future situation in which a possible benefit, interaction, environmental constraint, etc., is realized. Thus the client thumbnails emphasize hoped-for benefits of the VSF; the actor thumbnails suggest a few interaction variations anticipated for different stakeholders. The thumbnail scenarios generated in

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 10

this analysis are not yet design scenarios, they simply allow the analyst to begin to explore the space of user groups, motivations, and pragmatic constraints.

The CATWOE thumbnail scenarios begin the iterative process of identifying and analyzing the background, motivations, and preferences that different user groups will bring to the use of the target system. This initial picture will be elaborated throughout the development process, through analysis of both existing and envisioned usage situations.

CATWOE Element

V S F Element

Thumbnail Scenarios

Clients Students

Community members

A high school student learns about road-bed coatings from a retired civil engineer.

A busy housewife helps a middle school student organize her bibliographic information.

Actors Students

Teachers

Community members

A student imports an Excel spreadsheet containing her analysis of acid rainfall.

A teacher checks over the 10 projects underway by students in her class.

A retired pharmicist hears about the VSF over email and visits it for the first time.

Transformation Ongoing community- wide access to student projects

A student browses comments left by his friends, his father, and his former Boy Scout leader.

Weltanschauung Learning benefits from community involvement

After exhibiting a project on heat transfer, two students show up at a town meeting to discuss concerns about the new school.

Owners School district At a regional meeting, the school district reports on the number of visitors and comments contributed to the online fair.

Environment Computer labs

T1 lines to public meeting places (e.g., library)

Home modems

Several students stay after school and work on their separate projects together in the lab, chatting while they work.

A regular user of the town library sits down at the public terminal for the first time and is drawn into the online fair.

A mother logs on after dinner to visit with her friends and check in on her son’s project.

Table 3.1: Catwoe Elements and Stakeholder Thumbnail Scenarios for Science Fair

Observing Current Practice

The CATWOE analysis sets the scene for the study of current practice. With a shared understanding of major stakeholders and general project goals, the team can begin to analyze the activities that will be transformed by the system. In doing so, attention should be given to the needs and concerns of all parties. In the case of our VSF example, this implies analysis of students, teachers, community members, and the school organization. Although the CATWOE

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 11

analysis is intended as a scoping and planning aid, note that new stakeholders, motivations, or environmental factors may emerge through these studies of practice.

Preparing for Data Collection

The CATWOE analysis is just one initial activity that is useful in preparing for data collection. The process of identifying stakeholders and discussing orienting goals will raise many questions about the situation a team hopes to impact. For the VSF, the analysts might wonder what sorts of projects students currently develop and exhibit, what resources they draw on for project development, how the projects are exhibited and judged, how and when parents contribute to project work, and so on. These sorts of scoping discussions serve an orienting role, and for new teams can help group members learn about one another — personal background, interests, biases, as well as skills and aptitudes.

Guide for Interviewing Student-Participants in a Science Fair

Remember that our goal is to understand how and why students participate in the science fair. We want to know the things they do as part of the fair, and the sorts of resources (both physical and human) they use. We also want to learn something about the individuals we talk to—their history, and especially their use of or reactions to technology associated with science fairs.

Open-ended prompts (follow the interviewee’s lead):

How long have you been in science fairs; have you exhibited before?

How did you get involved in this year’s fair?

Tell me about your exhibit; can you show it to me? What did you do create it?

Did anyone else work with you on this project? How?

Tell me about the other people you have interacted with as part of the fair.

How will (or has) the judging take place?

What do you like about this science fair (or about your exhibit)? What are you unhappy with?

Specific things we want to know (ask directly if they’ve not yet been covered):

What technology (computer or otherwise) have you used in this project?

What technology would you have liked to use if it was available?

What background do you have in using computers?

How could an expert from the community contribute to your science project?

How could your parents contribute?

Can you imagine an online version of this science fair? What would it be like?

Table 3.2: Interviewing Guide for Field Study of Science Fair

In addition to developing a shared understanding of project scope, the team must consider its own organization. It will help of one member takes on a leadership role, ensuring that decisions are made about activities to be carried out, a schedule is constructed, everyone understands his or her responsibilities, and so on. It may even be useful to have two types of leaders, one who attends mostly to the content and results of the analysis task, another who makes sure that

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 12

everyone knows their role, stays on schedule, and so on. Other team members can take on the job of identifying and making contact with individuals will to be observed or interviewed. Others can focus on creating and assembling an interviewing guide and data capture tools.

An interviewing guide should be created that will support but not over-constrain how team members observe and question individuals in the work setting. It may be necessary to produce different guides for different stakeholder representatives. The guide should begin with an introduction that reminds the interviewer of what he or she hopes to accomplish in the questioning. Because the goal is to learn what the participants think about their own activities, the guide should not suggest specific and pointed questions early in the interview. Instead, begin with open-ended prompts that ascertain the interviewee’s general background and how they think about their work (“tell me about what you do”). List specific questions that emerge from group brainstorming at the end, so that the interviewers will be certain to address these issues if they are not raised by the interviewee. A guide for talking to student participants in a science fair appears in Table 3.2.

In addition to preparing an interviewing guide(s), the team must decide how to document their observations. If the work setting involves considerable physical manipulation of objects, a videotape may be helpful (though you should first check with participants to see if they are comfortable with this). Otherwise, a small tape recorder might be used to record conversations. In either case, plan in advance how you will use the recording equipment (e.g., where you will place the camera(s) or the microphone, how many tapes you will need), and be sure to get participants’ permission. It may also be useful to bring along a camera to capture interesting visual elements of the situation. Finally, one or more team members should be assigned the job of taking detailed written notes. Be very clear that all recordings, photos, or notes will be treated confidentially, reviewed and discussed only by your analysis team.

Observations, Interviews, and Artifacts

An important element of successful workplace studies is creating a comfortable relationship between the team and the workplace participants. In some settings, workers may have been selected by management and may be resentful at having to spend time with outsiders observing and interviewing them about their activities. In others, participation may be voluntary, but they may be intimidated by the thought of interacting with “technology-savvy” individuals. The team must be sensitive to social factors like this and focus first on establishing a friendly and non-judgmental tone to the exchange.

Figure 3.3, a photograph of a science fair in progess will appear here. It shows a student demonstrating an exhibit to a small group of visitors. It has not been included in this file because it is a color photo and makes the file too big! If you are interested, there is a link to it on the class website.

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 13

Whenever possible, visit a workplace instead of bringing participants to you. This puts the burden on you rather than the individuals you are studying, as well as giving you the opportunity to observe authentic work activities. If there is time and a participant is willing, observe her as she carries out one or more typical tasks. If you are collecting video or audio tape, remember to get the participant’s permission before turning it on, and be sure to identify the tape with date, time, place, and participant. Take notes conscientiously, writing down everything that happens, not just things that catch your attention. If something happens that you cannot understand, or if task details aren’t apparent (e.g., a computer activities that involve a sequence of data or procedures), interrupt briefly, just enough to get a summary of what is happening. After the observation period, interview the participant according to the guide developed in advance. Prior to launching into the material in your guide, ask the participant to comment on what they have just been doing; this provides a seamless transition from the observation period to the interview.

The photo in Figure 3.3 was taken during a visit to a science fair and documents interesting elements of current science fairs. For example, the student is using a computer as part of his exhibit, and is showing his project to several people at the same time. The observers appear to be a family group, with members ranging in age from a small child to an adult. Also in the room are other non-computer artifacts, posters that have been pinned up on the walls; in the picture it isn’t obvious what relation the posters have to the computer exhibits, but interviews with the students indicate that although only some projects have computer exhibits, all projects have a physical poster display.

The photo also displays various science fair artifacts. A science fair artifact is any resource that appears to play an important role in the task; it might be a computer program or data file, a poster, a registration or evaluation form, even a display stand. In some cases, these artifacts will be documented by videotapes or photographs; in other cases, the observation team will need to ask specifically for copies of relevant documents or other tools, or will simply make notes describing the object.

A process question for workplace studies concerns the amount of data to collect — how many visits and to how many different sites? Clearly this depends on the project’s scope, with larger projects needing more analysis. As a rule of thumb, we recommend that you collect at least one set of observations and/or interview for each stakeholder group from your CATWOE analysis. For the VSF project, this means analyzing the science fair activities and perspectives of students, teachers, community members (including parents), and school administration.

Summarizing Workplace Data

The field observations and interviews should increase the project team’s understanding of the backgrounds, expectations, and preferences of the stakeholders with respect to the technology that might be introduced into their workplace. The relevant data will have been obtained from a variety of sources: surveys or questionnaires administered in advance or after a visit, comments and behaviors of individuals, interviews, or public records of population characteristics. These

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 14

data should be compiled to develop user profiles that will guide subsequent scenario development. A sample set of profiles for the VSF stakeholders appears in Table 3.3.

VSF Stakeholder User Characteristics

Students Background : mixture of experience with computing applications, ranging from extensive use of computer games and several years of programming to minimal keyboarding skills and basic use of office applications. Moderate to extensive experience with Web browsing, email, and Internet chat.

Expectations: an online system should make construction of the exhibit easier and more fun. Likely to see the system as a variant of other (e.g., Web-based) hypermedia systems.

Preferences: Most comfortable with PC-Windows platform, from either school or home setting. Enjoy multimedia systems, even when slow over a phone line or other low band-width connection.

Community members Background : Bi-modal distribution with a few members having extensive computing experience through the work environment, and others with only modest (or no) exposure to email and a few Web applications. Many have visited their children’s exhibits (not always science) in the past.

Expectations: Many are unsure about how if at all they would contribute to creating a project, but able to imagine browsing exhibits online. No thoughts about encountering or talking to others while browsing the exhibits.

Preferences: those with background are comfortable with AOL and similar ISP environments, generally on PC-Windows platform. Less patience than students for “fancy” graphics or multimedia. Want guides or help screens to work through new applications or services; willing to read manuals.

Teachers Background : most are familiar with a range of computer applications, both Web-based browsing and discussion systems, as well as specialized courseware. A few have basic programming skills; most able to author documents in HTML and other hypermedia systems like PowerPoint.

Expectations: online system will allow them to draw in other experts to guide students in projects, decreasing the general need for teacher-student interaction, allowing them to focus on special needs. Most expect Web-based authoring and browsing combined with email discussions with outside experts.

Preferences: Mixture of PC and Mac users. Strong concerns about access rights and about ability to get overview information. Willing to work from online or written guides or reference material. Want example (starter) projects.

School administrators Background : Familiar with Web-browsing, email, and standard office applications, especially word-processing and spreadsheet functions.

Expectations: online system will increase visibility of science fair, create better connections between the school and the community. Emphasis will be on conveying to community the interesting things that students are doing. Like teachers, expect such a system to involve combination of Web and email.

Preferences: Mixture of PC and Mac users. Concerned that system is state-of- the-art and attractive; will want to print colorful examples and summaries of the online materials to share with community groups and agencies.

Table 3.3: Stakeholder Profiles for the VSF

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 15

A second summary should be prepared for tasks that were observed or discussed. To do this, the team can begin with a list of tasks relevant to each stakeholder. For particularly complex or problematic tasks, it may be useful to develop a hierarchical task analysis to represent documenting a finer level of detail. A summary of the tasks of VSF stakeholders is in Table 3.4, with examples of hierarchical task analysis in Figure 3.4 (Figure 3.4a shows exhibit construction, Figure 3.4b shows exhibit judging).

VSF Stakeholder Science Fair Tasks Observed or Discussed

Students Reviewing participation requirements; Proposing a project; Carrying out the project; Constructing an exhibit; Demonstrating the project

Community members Browsing projects at a fair; Interacting with students at their exhibits

Community members acting as judges

Volunteering to be a judge; Studying the evaluation form; Evaluating a specific project; Developing and reporting summary results

Teachers Helping a student refine a proposal project; Providing pointers to resources and other information; Critiquing a student’s project in progress; Helping a student create an exhibit

School administrators Recruiting volunteers to judge projects; Summarizing participation in fair; Highlighting winning projects in annual report; Specifying resources needed for next year’s fair; Acknowledging student and judge participation

Table 3.4: Science Fair Tasks Carried out by Stakeholders

In addition to field notes and interview data, the requirements team may have collected task artifacts—data files, forms, software, brochures, and so on. These artifacts can help to document the information needs of the tasks they serve. A form used to purchase something indicates what information is needed to specify the product, the buyer, the form of payment, and so on. A poster used to advertise an event shows what features of an event are important to the intended audience. The labels on file folders suggest categorical information that is used to organize task materials. Each artifact collected should be examined for such insights. Figure 3.5 shows two artifacts from a science fair, a publicity poster and a judging form.

Other artifacts from this domain might include the registration form used by students to enter their projects, newspaper notices recruiting volunteers to serve as judges and later announcing the winners, the instructions provided to judges and to student exhibitors, the exhibits themselves, the prize ribbons, even the thank-you notes sent by fair organizers to the volunteers helping to set up and judge the exhibits. The role of task artifacts in current practice can be summarized as shown in Table 3.5 for the poster and the judging form — both the information needs and the science fair procedures implied by these two documents have been listed. From the publicity perspective, we assume that the poster emphasizes what the organizers believes are the most important characteristics of the event to possible attendees. From the perspective of exhibit evaluation, the judges’ form likewise highlights the characteristics thought to determine the quality of the exhibits. Note that the insights garnered by studying these artifacts may or may not match those obtained through interviews or observations; as noted in the section on Understanding

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 16

Figure 3.4a: Hierarchical task analysis for constructing an exhibit.

0. Construct exhibit

1. Analyze project content 2. Develop an exhibit plan 3. Lay out exhibit elements

3.1 Center title and abstracts at top of

display space

3.2 Position graphical and

physical elements

3.3 Place summary and sources at

lower right

1.1 List elements of overall project

structure

1.2 Summarize exhibit content for

each element

2.1 Assess exhibit space available

2.2 Write title, credits and abstract

2.3 Develop effective visual components

2.4 Develop explanatory text

plan 0: do 1 - 3, with iteration as necessary

plan 1: do 1.1 , then do 1.2

2.3.1 Collect key data graphs or

charts

2.3.2 Collect photos or videos of apparatus

2.3.3 Collect relevant physical models

2.3.4 Select elements

with greatest impact

plan 2: do 2.1 - 2.2, then interleave 2.3 - 2.4 until done

plan 2.3: do 2.3.1 - 2.3.3 as relevant, do 2.3.4 , then do 2.3.5 iteratively until done

2.3.5 Develop caption for

each graphic or model

2.4.1 Write a summary and conclusions

2.4.2 List bibliography

and other sources

plan 2.34 do 2.4.1 - 2.4.2

3.4.1 Place captions near

associated visuals

3.4.2 Insert additional

explanatory text

3.4 Interleave supporting textual

material

plan 3.4: do 3.4.1 then 3.4.2

plan 3: do 3..1

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 17

Figure 3.4b: Hierarchical task analysis for judging an exhibit.

0. Judge an exhibit

1. Browse exhibit structure 2. Study exhibit in detail 4. Complete judging report

4.1 Award points for significance of problem studied

4.3 Award points for quality of project

presentation

1.1 Read project title and

abstract

1.2 Scan visual and physical

elements

2.1 Read captions and study details of visual elements

2.2 Examine and manipulate physical

models

2.3 Identify and follow logic of scientific methods used

2.4 Analyze conclusions with respect to

methods and results

plan 0: do 1 - 4

plan 1: do 1.1 - 1.3

4.4 Review, sign and submit project

evaluation form

plan 4.3: do 4.3.1 - 4.3.3 together

plan 3: do 3.1 - 3.5

3. Interview exhibit author

1.3 Skim conclusions and

bibliography

3.1 Listen to project

summary

3.2 Probe problem

understanding

3.3 Probe use of scientific method

3.4 Ask about alternative

interpretations

3.5 Complement content and presentation

4.2 Award points for quality of project

content

4.2.1 Award points for method

soundness

4.2.2 Award points for originality

4.2.3 Award points for overall

coherence

4.2.4 Award points for use of

resources

4.3.1 Award points for visual

details

4.3.2 Award points for

supporting text

plan 4: do 4.1 - 4.3 together, then do 4.4

plan 2: do 2.1 - 2.4 together

plan 4.2: do 4.2.1 - 4.2.4 together

4.3.3 Award points for

layout

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 18

Users, much of what participants know or experience in a situation is based on tacit knowledge, or is the result of other stakeholders’ views and concerns.

Figure 3.5: Sample artifacts from a science fair.

Science Fair Artifact

Implied Information Needs and Procedures

Fair publicity poster Information: When and where fair is held; sponsoring organization; time when winning exhibits announced; sample projects; contact information

Procedures: fair lasts about 2 hours; judging takes about 1.5 hours; projects take up to 6 months to complete; exhibits are entered and judged in three age-level groupings

Judging form Information: judging may be age-level specific; exhibits are judged on three major dimensions; quality is a complex judgement broken into sub-categories; Ms. Czerny is the head judge who compiles the separate results

Procedures: personal contact with students helps to assess project significance and quality; exhibits are judged in about 15 minutes; forms are not to be submitted until all judging is completed

Table 3.5: Information and Procedures implied by Poster and Judges’ Form

The analysis of stakeholder groups, tasks, and artifacts, focuses on individual elements of the current situation. This helps a team organize their observations, but can also direct attention away from the social context in which the activities take place, the network of interdependencies

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 19

among stakeholders and their roles in the workplace. One way to summarize this is to create a stakeholder diagram that conveys how the stakeholders are interdependent on one another. Figure 3.6 presents such a diagram for the science fair. For example, we see that one role of students in the science fair is to create projects that community members will browse and perhaps judge. These relationships can then be analyzed to understand the impact one group of participants has on others. For instance the number of projects developed for the fair will have an impact on the community members who judge the exhibits — perhaps affecting how many judges will be needed, as well as the challenge and satisfaction of the evaluation process.

Another technique for grasping the big picture is to collect together related observations or problems into a set of workplace themes. Different stakeholders will focus on different aspects of the current situation (e.g., based on their particular motivations or backgrounds), but issues raised by different stakeholders (or at different times by the same stakeholder) will be related. To find a useful set of themes, write interesting comments or observations on index cards or post-it notes, and then carry out a collaborative grouping exercise where team members search for related points. In some cases an issue will fit in more than one category; this is fine, simply make some indication that multiple copies have been made. Once a theme has been suggested, test its usefulness by trying to name it — if you find yourself with too many “Miscellaneous” or “Other Problems” groups, your work is not done yet! Beyers and Holtzblatt (1998) use such techniques extensively to create affinity diagrams, which are then reviewed, discussed, and elaborated with participating stakeholders. Figure 3.7 shows some themes identified in analysis of current science fairs.

Figure 3.6: Roles and relations of stakeholders in science fair.

Student exhibitors

Teachers

Community members

School administration

advertise fair to

browse exhibits of; (may) judge exhibits of

provide resources to;

summarize results of;

acknowledge participation by

recruit volunteers from;

acknowledge participation by

create projects for viewing by;

guide activities of

interact with

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 20

Figure 3.7: Themes summarizing issues raised in science fair study.

Developing Requirements Scenarios

User, task, artifact, roles, and theme analyses are the scaffolding from which requirements scenarios are synthesized. The goal of requirements scenarios is to express — in a condense and evocative fashion — the key understandings that the team has gained about the workplace. Other members of the development team (e.g., software engineers) should be able to read the scenarios and appreciate many of the work-related issues that your analysis has uncovered. Look across the perspectives generated by the user, task, artifact, and social foci and extract the issues that have the strongest implications (either opportunities or constraints) for design. Weave these issues together to build illustrative episodes of current practice.

Exhibit Construction Exhibit Judging Fair Attendance

Often must combine work done at home and at school or in special facilities

Some projects must crowds in all the visuals, others end up with extra space

Dynamic elements (e.g. videos) hard to set up and manage

Limiting the time for judging leads to rushing and feelings of stress

Hard to compare exhibits separated by more than a few feet

Manual compilation of final results is tedious and error- prone

Difficult to get the word out to people who are not parents of students

Evening exhibit hours compete with other family events or obligations

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 21

Requirements Scenarios from the Science Fair

1) Ms. Smith helps Jeff plan his exhibit on heat transfer.

Ms. Smith worked late on Monday evening so she’d have time to on Tuesday with Jeff, one of her top physics students, to help him plan his science fair exhibit. When she got to the lab, Jeff was already there and had blocked off a 6-foot rectangle on a lab table, simulating the space he’d get at the gym. He also had brought a sample poster board to use in laying out the three walls of the exhibit. They spent the first 20 minutes just listing possible project elements. Ms. Smith was impressed with the range of content Jeff had developed, including an extensive list of URLs on heat transfer, several charts graphing the effects of three different window coatings, and an Authorware simulation of heat diffusion. This last piece of the project concerned her, as last year there had been no computer hook-ups in the gym. But as usual, Jeff had already checked into this, and had gotten special permission to have this in his display, as long as he took responsibility for the set-up. As Ms. Smith checked through the visuals, she noticed that the captions for the color charts used a different font than those for the black-and-white drawings of his apparatus; Jeff explained that he had printed the former on his personal color printer at home, which has a different font set. Ms. Smith knew that the judges would notice even details like this, so she advised him to re-print the black-and-white drawings at home. They spent the rest of the time selecting and laying out the visuals that would make best use of the limited space.

2) .Mrs Sampson decides to go to the science fair.

Mrs. Sampson’s neighbor Jeff was in the science fair for the third time this year, and she really wanted to go, especially now that her seventh-grade daughter Erin seemed to be developing an interest in science. Jeff had mentioned the fair date, but she had forgotten about it until she saw the poster at Kroger’s She mentioned her plan to her husband, who immediately reminded her that this overlaps with Billy’s basketball game. They agreed to split their time that evening. On the night of the fair, her 5-year-old Christie decided she’d rather go with Mom than Dad and Billy, so the three of them headed off. But as soon as they got in the car, Christie started complaining that she never gets to do anything special on her own. Mrs. Sampson and Erin ignored her as much as they could, and started talking about who else might be exhibiting projects.

3) Jeff demonstrates his exhibit to Mrs. Sampson and her two daughters.

When the Sampsons arrived at the fair, there were only a few other people around. They saw their neighbors, Jeff’s parents, and a few other people they didn’t recognize. Jeff’s Mom chatted with them briefly, asking about Erin’s summer plans, then summarizing some highlights of exhibits she’d seen so far. The Sampsons began browsing, starting on the right-hand side and working their way around, mostly just looking quickly and moving on, so they’d have time to see as much as possible. When they got to Jeff’s exhibit, they stopped to talk to him, and he gave them his overview. Erin was very interested in his charts, and wanted to know just how he had gotten the data and graphed it, but Christie quickly became bored and started poking around at the computer on the table. Mrs. Sampson saw the list of URLs and thought Christie might want to look into them, so started copying them down; when Jeff saw this, he offered to print them out later and give her a copy. After the overview, Jeff started to show them the animation, but found that Christie had managed to crash the computer. While they were waiting for it to re-boot, Christie wondered whether there were any other exhibits related to heat transfer or building construction, but Jeff hadn’t had time to look around, so didn’t know.

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 22

4) Alisa judges the high school physics projects at the science fair.

For the last three years, Alisa—a retired civil engineer—has been a judge at the county science fair, so when the organizers contacted her, she readily agreed to do it again. In past years, she had occasionally been able to get advance information about the projects assigned to her, but this year the organizers had collected only titles and authors in advance. As in the past, she saw that she’d been given the high school physics projects, and knew she’d have to work fast to get all five evaluated in the 90 minutes allotted. On the night of the fair, Alisa arrived promptly at 7pm, picked up her forms and began the process of studying the exhibits and interviewing the students. Her previous experience helped her to make the points assignment judgements, but as usual she found it hard to compare projects even within her set of five. At one point, she found she needed to evaluate two very nice projects in parallel, running back and forth comparing the details of the visuals and models, and annotating her scores with relative comparisons. She finally signed her forms and handed them in—she could tell she was almost the last to finish from the large stack of papers Ms. Czerny and her assistants were already compiling.

5) Superintendent Carlisle reports on th 1999 science fair to the school board.

School superintendent Mark Carlisle had heard wonderful reports about this year’s science fair, so he decided to highlight it in next month’s school board meeting. He wanted to do more than acknowledge the winners and the volunteers—he hoped that by giving the school board examples of the great work the students had done, he could make a case for increasing the resources allotted for extra-curricular activities such as this. He contacted Ms. Czerny, and asked her to collect sample materials from the best projects, so that he could construct a slide presentation. She and a colleague spent the next two weeks tracking down the winning authors and finding high quality visuals. Carlisle then cycled through these to find just the few that he could include in his 10 minute presentation, but he brought paper copies of many others in case the school board was interested. He highlighted Jeff’s exhibit, noting that he had set up his own computer for demonstration, and pointing to several other exhibits that would have been much enhanced by such technology. Though the pitch was well-received, by the time that budget discussions took place the board members had forgotten many details and were reluctant to increase the funds for technology support of events like the science fair.

Table 3.6: Requirements Scenarios from the Science Fair Analysis

Scenario writing is a creative act involving judgment and the integration of multiple issues. A scenario may narrate the experiences of several stakeholders if collaborative activity is involved, or it may focus on an individual. The scope of a scenario is meaningful activity; every scenario should be motivated by and contribute to the overall work context. Begin with one scenario for each stakeholder, focusing on a central (or particularly problematic) activity for that group. Develop a scenario around this activity, using the themes analysis to insert an overall “message”, and then elaborate with issues raised by artifact analysis, role relationships, and so on. Some issues will fit naturally into many scenarios, suggesting an overarching concern to address in design. After developing one scenario for each stakeholder, review your workplace analyses and find issues not yet covered. Are any remaining issues important enough to add into an existing scenario or create a new one? If so, continue to elaborate the set. If it is possible, invite your stakeholders to participate in scenario construction, review, or elaboration. In the end, not all issues will be covered by the scenario set, but the important ones will be. Table 3.6 lists some requirements scenarios generated for the science fair problem.

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 23

Several points are worth noting about these scenarios. They deliberately reuse actors and artifacts (e.g., Jeff and his exhibit, the head judge). This adds to the overall coherence of the scenario set: the analysis of Jeff’s project that is appropriate for the exhibiting scenario must also make sense in the context of the planning, mentoring, judging, and archiving activities. It also encourages a more expansive analysis of the people and artifacts in the domain, by considering their contributions to more than one high-level activity. Of course, this comes with a cost of not illustrating the contributions of contrasting people and artifacts—perhaps a student who wants a computer demonstration but is unable to set it up, or a judge who had no prior experience and was unable to complete her work on time. It is in this sense that requirements scenarios should be seen as only as an illustrative set of “stakes in the ground”. They are suggestive, not exhaustive.

The scenarios also include details about participants’ real world situations that do not contribute to the science fair activity itself. It is important to express concretely who the actors are and to convey the many forces that are influencing their behavior in the situation. The teacher helping Jeff is very busy; this is typical but has nothing to do with the science fair itself. Mrs. Sampson’s family life, her problems with her young daughter, have no direct impact on the fair or its operation. However, these details provide bits of context that help the analyst think about these actors’ motivations and experience. Specific details like these also encourage analysts to think about other situations in which different factors are in play, for example a case where Jeff’s neighbor lives alone and has no other obligations. This is one of the important benefits of writing and sharing scenarios in requirements analysis.

Analyzing a Scenario’s Claims

Writing a scenario is a creative act, but it is an act that is informed by many things. The studies of the workplace educate the requirements team, and scenario creation enables the team to synthesize and express their new knowledge. Implicit in these analysis and design activities are the tradeoffs present in the problem situation — features that can be understood to have both positive and negative consequences for use. These tradeoffs may not be directly conveyed in the requirements scenarios, but taking the time to consider them can help you begin to reason about how you might transform the current situation. In general the design goal will be to increase (or add new) positive consequences and to decrease (or remove) negative consequences.

SBUE employs claims analysis to reason about tradeoffs in observed or imagined usage situations. Analyzing a claim begins with the identification of an “interesting feature” of a scenario, some aspect of an activity that seems to have one or more important impacts on task participants. Because we are interested primarily in information technology, we tend to focus on features of work-related artifacts, particularly artifacts that are or could be computer-based. Example features from the science fair artifacts might include the size of the space allotted for individual exhibits, the physical lay-out of exhibits in the hall, or the science fair information advertized on the poster.

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 24

Scenario Feature P o s s i b l e U p s i d e s ( + ) o r D o w n s i d e s ( - ) o f t h e F e a t u r e

The fixed area allotted to each project exhibit

(from Scenarios 1 & 3)

+ constrains the complexity of individual exhibits

+ simplifies planning and layout of the total space available

+ creates a grid that can organize browsing and judging activities

- but many exhibits will have genuinely different needs for space

- but too regular a grid may make the layout seem routine and boring

- but physical constraints will limit the grouping possibilities for related exhibits

The date, time, and location advertized on the science fair poster

(from Scenario 2)

+ makes the event seem more concrete (tied to a specific place and point in time)

+ sets clear boundaries on students’ exhibition responsibilities

+ enables planned or opportunistic simultaneous visitors

- but may lead some to skip the fair due to competition with other events

Handwritten judges’ forms

(from Scenario 4)

+ are a familiar technology for recording information and decisions

+ support personalized handwritten annotations and qualifications

+ offer a natural authentication function (i.e., by signature)

- but may be hard to modify in response to unexpected issues or characteristics

- but will lead to significant paper-processing requirements at the end of judging

Physical exhibits

(from Scenarios 1, 3, & 5)

+ encourage the use of many diverse exhibit elements

+ allow for direct interaction and engagement by visitors and judges

- but abstract or dynamic elements may be difficult to express in physical form

- but large physical elements may be difficult to construct or display

- but physical exhibits may be difficult to archive, transport, copy, or share

Table 3.7: Claims from the Science Fair Requirements Scenarios

The second piece of claims analysis is hypothesizing the consequences a feature has for the stakeholder(s) in the scenario. How does the feature make the activity easier? Harder? More or less pleasant or satisfying? Some consequences will be readily apparent in a scenario as written. Sometimes a set of consequences can be seen in different scenarios touching on the same feature of the workplace. Some may not be part of a scenario at first but are added as an elaboration after thinking about some other scenario. Yet others will be part of some other story, a scenario version where things work out a bit differently than imagined. These are the hardest consequences to realize, but are critical to include. They reflect a kind of “what if” reasoning about the situation that expands the scope of the analysis as well as encouraging analysts to look ahead to situations that a new system might support.

An important feature of claims analysis is that both positive and negative consequences (informally termed the “upsides” and “downsides”) are considered. When examining a current situation, it is easy to focus on just the problems imposed by current technology; later on, when analyzing the impact of new technology, it will be easy to focus on just the advantages provided by the

DRAFT: PLEASE DO NOT CITE OR CIRCULATE WITHOUT PERMISSION

SBUE—Chapter 3 25

new system. However we cannot emphasize enough that all design involves tradeoffs—you rarely get something for nothing. By imposing the discipline of documenting upsides and downsides for every claim, you’ll be more likely to understand and address both sides of a design issue.

Table 3.7 summarizes claims analyzed for several VSF requirements scenarios. Each claim considers a range of possible positive and negative consequences of a feature impacting one or more scenarios. Some consequences are evident in the narratives as written: Jeff and his teacher are well-aware of the constraints of the exhibit space; Alisa is able to annotate her form with notes concerning the two projects of similar quality; the Sampson family is clearly engaged by the physical characteristics of Jeff’s exhibit. Other consequences emerge from what-if reasoning inspired by these scenarios: Jeff is able to select project elements that fit into the space allowed, but some students may find they must leave out interesting elements; the 2-hour time period of the fair tells the Sampsons when to visit, and at the same time puts well-defined boundaries on the responsibilities of student authors managing their own busy schedules; Ms. Czerny was able to collect a nice set of samples for the superintendent’s presentation, but some projects might not have been saved in a form suitable for this purpose. In this sense, a claims analysis allows a requirements analysts to point to scenario elaborations without developing the corresponding narratives. More importantly, it documents the analysts’ conclusions about tradeoffs in the current situation that should be addressed by the new design.

Scenarios and Claims as Requirements

In what sense do scenarios and claims convey requirements? Clearly a set of requirements scenarios is not a specification. It captures insights gleaned about the current situation. Subsequent design reasoning will develop a response to this analysis, creating and refining a design specification that is eventually implemented as the final system. The requirements expressed by the scenarios relate to the needs and possibilities suggested by analysis of current practice: The scenarios narrate instances of typical or critical goals and activities. Specific features of these activities are seen to have impact, and associated claims analyses articulate what might be good or bad about such features. The implicit assumption is that the design process will respond to these needs and possibilities, hopefully maintaining or improving on the positive characteristics and diminishing or removing the negative.

This textbook deliberately presents a simplified view of SBUE. In practice analysis and design will take place in a tightly interleaved fashion: As soon as requirements analysts recognize a problem in the current situation, they will begin to develop possible design approaches to solving the problem. There is some evidence of this in the VSF examples, because the analysis focused almost immediately on the problems and opportunities most likely to be influenced by online activities (e.g., eliminating scheduling and space constraints). At the same time, because the entire process is iterative, we know that the scenarios and claims developed as the result of this initial requriements analysis are only a first pass at understanding the problem. The development of requirements will continue as design ideas emerge and are considered through new scenarios and tradeoff analyses.

The drumbeat for improved software quality began in earnest as softwarebecame increasingly integrated in every facet of our lives. By the 1990s,major corporations recognized that billions of dollars each year were be- ing wasted on software that didn’t deliver the features and functionality that were promised. Worse, both government and industry became increasingly concerned that a major software fault might cripple important infrastructure, costing tens of billions more. By the turn of the century, CIO Magazine [Lev01] trumpeted the headline, “Let’s Stop Wasting $78 Billion a Year,” lamenting the fact that “American businesses spend billions for software that doesn’t do what it’s sup- posed to do.” InformationWeek [Ric01] echoed the same concern:

Despite good intentions, defective code remains the hobgoblin of the software indus-

try, accounting for as much as 45% of computer-system downtime and costing U.S.

companies about $100 billion last year in lost productivity and repairs, says the

Standish Group, a market research firm. That doesn’t include the cost of losing angry

customers. Because IT shops write applications that rely on packaged infrastructure

software, bad code can wreak havoc on custom apps as well. . . .

Just how bad is bad software? Definitions vary, but experts say it takes only three or

four defects per 1,000 lines of code to make a program perform poorly. Factor in that most

programmers inject about one error for every 10 lines of code they write, multiply that by

the millions of lines of code in many commercial products, then figure it costs software

vendors at least half their development budgets to fix errors while testing. Get the picture?

398

C H A P T E R

14 QUALITYCONCEPTS K E Y C O N C E P T S cost of quality . .407 good enough . . .406 liability . . . . . .410 management actions . . . . . . .411 quality . . . . . . .399 quality dilemma . . . . . .406 quality dimensions . . . .401 quality factors .402 quantitative view . . . . . . . .405 risks . . . . . . . .409 security . . . . . .410

What is it? The answer isn’t as easy as you might think. You know quality when you see it, and yet, it can be an elusive thing to define. But for com-

puter software, quality is something that we must define, and that’s what I’ll do in this chapter.

Who does it? Everyone—software engineers, managers, all stakeholders—involved in the software process is responsible for quality.

Why is it important? You can do it right, or you can do it over again. If a software team stresses quality in all software engineering activities, it reduces the amount of rework that it must do. That results in lower costs, and more importantly, improved time-to-market.

Q U I C K L O O K

What are the steps? To achieve high-quality software, four activities must occur: proven soft- ware engineering process and practice, solid project management, comprehensive quality control, and the presence of a quality assurance infrastructure.

What is the work product? Software that meets its customer’s needs, performs accurately and reliably, and provides value to all who use it.

How do I ensure that I’ve done it right? Track quality by examining the results of all quality control activities, and measure quality by exam- ining errors before delivery and defects released to the field.

pre75977_ch14.qxd 11/27/08 5:51 PM Page 398

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 399

In 2005, ComputerWorld [Hil05] lamented that “bad software plagues nearly every organization that uses computers, causing lost work hours during computer down- time, lost or corrupted data, missed sales opportunities, high IT support and mainte- nance costs, and low customer satisfaction. A year later, InfoWorld [Fos06] wrote about the “the sorry state of software quality” reporting that the quality problem had not gotten any better.

Today, software quality remains an issue, but who is to blame? Customers blame developers, arguing that sloppy practices lead to low-quality software. Developers blame customers (and other stakeholders), arguing that irrational delivery dates and a continuing stream of changes force them to deliver software before it has been fully validated. Who’s right? Both—and that’s the problem. In this chapter, I consider soft- ware quality as a concept and examine why it’s worthy of serious consideration whenever software engineering practices are applied.

1 4 . 1 W H AT I S Q U A L I T Y ? In his mystical book, Zen and the Art of Motorcycle Maintenance, Robert Persig [Per74] commented on the thing we call quality:

Quality . . . you know what it is, yet you don’t know what it is. But that’s self-contradictory.

But some things are better than others; that is, they have more quality. But when you try

to say what the quality is, apart from the things that have it, it all goes poof! There’s noth-

ing to talk about. But if you can’t say what Quality is, how do you know what it is, or how

do you know that it even exists? If no one knows what it is, then for all practical purposes

it doesn’t exist at all. But for all practical purposes it really does exist. What else are the

grades based on? Why else would people pay fortunes for some things and throw others

in the trash pile? Obviously some things are better than others . . . but what’s the better-

ness? . . . So round and round you go, spinning mental wheels and nowhere finding any-

place to get traction. What the hell is Quality? What is it?

Indeed—what is it? At a somewhat more pragmatic level, David Garvin [Gar84] of the Harvard Busi-

ness School suggests that “quality is a complex and multifaceted concept” that can be described from five different points of view. The transcendental view argues (like Persig) that quality is something that you immediately recognize, but cannot explic- itly define. The user view sees quality in terms of an end user’s specific goals. If a product meets those goals, it exhibits quality. The manufacturer’s view defines qual- ity in terms of the original specification of the product. If the product conforms to the spec, it exhibits quality. The product view suggests that quality can be tied to inher- ent characteristics (e.g., functions and features) of a product. Finally, the value-based view measures quality based on how much a customer is willing to pay for a prod- uct. In reality, quality encompasses all of these views and more.

Quality of design refers to the characteristics that designers specify for a product. The grade of materials, tolerances, and performance specifications all contribute to the quality of design. As higher-grade materials are used, tighter tolerances and

What are the different ways in which quality can be viewed?

pre75977_ch14.qxd 11/27/08 5:51 PM Page 399

greater levels of performance are specified, the design quality of a product increases, if the product is manufactured according to specifications.

In software development, quality of design encompasses the degree to which the design meets the functions and features specified in the requirements model. Quality of conformance focuses on the degree to which the implementation follows the design and the resulting system meets its requirements and performance goals.

But are quality of design and quality of conformance the only issues that software engineers must consider? Robert Glass [Gla98] argues that a more “intuitive” rela- tionship is in order:

user satisfaction ! compliant product " good quality " delivery within budget and schedule

At the bottom line, Glass contends that quality is important, but if the user isn’t satisfied, nothing else really matters. DeMarco [DeM98] reinforces this view when he states: “A product’s quality is a function of how much it changes the world for the better.” This view of quality contends that if a software product provides substantial benefit to its end users, they may be willing to tolerate occasional reliability or per- formance problems.

1 4 . 2 S O F T WA R E Q U A L I T Y Even the most jaded software developers will agree that high-quality software is an important goal. But how do we define software quality? In the most general sense, software quality can be defined1 as: An effective software process applied in a manner that creates a useful product that provides measurable value for those who produce it and those who use it.

There is little question that the preceding definition could be modified or extended and debated endlessly. For the purposes of this book, the definition serves to emphasize three important points:

1. An effective software process establishes the infrastructure that supports any effort at building a high-quality software product. The management aspects of process create the checks and balances that help avoid project chaos—a key contributor to poor quality. Software engineering practices allow the developer to analyze the problem and design a solid solution—both critical to building high-quality software. Finally, umbrella activities such as change management and technical reviews have as much to do with quality as any other part of software engineering practice.

2. A useful product delivers the content, functions, and features that the end user desires, but as important, it delivers these assets in a reliable, error-free

400 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

uote:

“People forget how fast you did a job— but they always remember how well you did it.”

Howard Newton

1 This definition has been adapted from [Bes04] and replaces a more manufacturing-oriented view presented in earlier editions of this book.

pre75977_ch14.qxd 11/27/08 5:51 PM Page 400

way. A useful product always satisfies those requirements that have been explicitly stated by stakeholders. In addition, it satisfies a set of implicit requirements (e.g., ease of use) that are expected of all high-quality software.

3. By adding value for both the producer and user of a software product, high- quality software provides benefits for the software organization and the end- user community. The software organization gains added value because high-quality software requires less maintenance effort, fewer bug fixes, and reduced customer support. This enables software engineers to spend more time creating new applications and less on rework. The user community gains added value because the application provides a useful capability in a way that expedites some business process. The end result is (1) greater software product revenue, (2) better profitability when an application supports a business process, and/or (3) improved availability of information that is crucial for the business.

14.2.1 Garvin’s Quality Dimensions

David Garvin [Gar87] suggests that quality should be considered by taking a multidi- mensional viewpoint that begins with an assessment of conformance and termi- nates with a transcendental (aesthetic) view. Although Garvin’s eight dimensions of quality were not developed specifically for software, they can be applied when soft- ware quality is considered:

Performance quality. Does the software deliver all content, functions, and features that are specified as part of the requirements model in a way that provides value to the end user?

Feature quality. Does the software provide features that surprise and delight first-time end users?

Reliability. Does the software deliver all features and capability without failure? Is it available when it is needed? Does it deliver functionality that is error-free?

Conformance. Does the software conform to local and external software standards that are relevant to the application? Does it conform to de facto design and coding conventions? For example, does the user interface con- form to accepted design rules for menu selection or data input?

Durability. Can the software be maintained (changed) or corrected (debugged) without the inadvertent generation of unintended side effects? Will changes cause the error rate or reliability to degrade with time?

Serviceability. Can the software be maintained (changed) or corrected (debugged) in an acceptably short time period? Can support staff acquire all information they need to make changes or correct defects? Douglas Adams [Ada93] makes a wry comment that seems appropriate here: “The difference

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 401

pre75977_ch14.qxd 11/27/08 5:51 PM Page 401

between something that can go wrong and something that can’t possibly go wrong is that when something that can’t possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.”

Aesthetics. There’s no question that each of us has a different and very subjective vision of what is aesthetic. And yet, most of us would agree that an aesthetic entity has a certain elegance, a unique flow, and an obvious “presence” that are hard to quantify but are evident nonetheless. Aesthetic software has these characteristics.

Perception. In some situations, you have a set of prejudices that will influ- ence your perception of quality. For example, if you are introduced to a soft- ware product that was built by a vendor who has produced poor quality in the past, your guard will be raised and your perception of the current soft- ware product quality might be influenced negatively. Similarly, if a vendor has an excellent reputation, you may perceive quality, even when it does not really exist.

Garvin’s quality dimensions provide you with a “soft” look at software quality. Many (but not all) of these dimensions can only be considered subjectively. For this reason, you also need a set of “hard” quality factors that can be categorized in two broad groups: (1) factors that can be directly measured (e.g., defects uncovered dur- ing testing) and (2) factors that can be measured only indirectly (e.g., usability or maintainability). In each case measurement must occur. You should compare the software to some datum and arrive at an indication of quality.

14.2.2 McCall’s Quality Factors

McCall, Richards, and Walters [McC77] propose a useful categorization of factors that affect software quality. These software quality factors, shown in Figure 14.1, focus on three important aspects of a software product: its operational characteris- tics, its ability to undergo change, and its adaptability to new environments.

Referring to the factors noted in Figure 14.1, McCall and his colleagues provide the following descriptions:

Correctness. The extent to which a program satisfies its specification and fulfills the

customer’s mission objectives.

Reliability. The extent to which a program can be expected to perform its intended func-

tion with required precision. [It should be noted that other, more complete definitions of

reliability have been proposed (see Chapter 25).]

Efficiency. The amount of computing resources and code required by a program to

perform its function.

Integrity. Extent to which access to software or data by unauthorized persons can be

controlled.

Usability. Effort required to learn, operate, prepare input for, and interpret output of a

program.

402 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

pre75977_ch14.qxd 11/27/08 5:51 PM Page 402

Maintainability. Effort required to locate and fix an error in a program. [This is a very

limited definition.]

Flexibility. Effort required to modify an operational program.

Testability. Effort required to test a program to ensure that it performs its intended

function.

Portability. Effort required to transfer the program from one hardware and/or software

system environment to another.

Reusability. Extent to which a program [or parts of a program] can be reused in other

applications—related to the packaging and scope of the functions that the program

performs.

Interoperability. Effort required to couple one system to another.

It is difficult, and in some cases impossible, to develop direct measures2 of these quality factors. In fact, many of the metrics defined by McCall et al. can be measured only indirectly. However, assessing the quality of an application using these factors will provide you with a solid indication of software quality.

14.2.3 ISO 9126 Quality Factors

The ISO 9126 standard was developed in an attempt to identify the key quality attributes for computer software. The standard identifies six key quality attributes:

Functionality. The degree to which the software satisfies stated needs as indicated by the following subattributes: suitability, accuracy, interoperability, compliance, and security.

Reliability. The amount of time that the software is available for use as indi- cated by the following subattributes: maturity, fault tolerance, recoverability.

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 403

PRODUCT OPERATION

PRODUCT TRANSITIONPRODUCT REVISION

Correctness Usability Efficiency Reliability Integrity

Maintainability Flexibility Testability

Portability Reusability Interoperability

FIGURE 14.1 McCall’s software quality factors

uote:

“The bitterness of poor quality remains long after the sweetness of meeting the schedule has been forgotten.”

Karl Weigers (unattributed quote)

2 A direct measure implies that there is a single countable value that provides a direct indication of the attribute being examined. For example, the “size” of a program can be measured directly by counting the number of lines of code.

pre75977_ch14.qxd 11/27/08 5:51 PM Page 403

Usability. The degree to which the software is easy to use as indicated by the following subattributes: understandability, learnability, operability.

Efficiency. The degree to which the software makes optimal use of system resources as indicated by the following subattributes: time behavior, resource behavior.

Maintainability. The ease with which repair may be made to the software as indicated by the following subattributes: analyzability, changeability, stability, testability.

Portability. The ease with which the software can be transposed from one environment to another as indicated by the following subattributes: adapt- ability, installability, conformance, replaceability.

Like other software quality factors discussed in the preceding subsections, the ISO 9126 factors do not necessarily lend themselves to direct measurement. However, they do provide a worthwhile basis for indirect measures and an excellent checklist for assessing the quality of a system.

14.2.4 Targeted Quality Factors

The quality dimensions and factors presented in Sections 14.2.1 and 14.2.2 focus on the software as a whole and can be used as a generic indication of the quality of an application. A software team can develop a set of quality characteristics and associ- ated questions that would probe3 the degree to which each factor has been satisfied. For example, McCall identifies usability as an important quality factor. If you were asked to review a user interface and assess its usability, how would you proceed? You might start with the subattributes suggested by McCall—understandability, learnability, and operability—but what do these mean in a pragmatic sense?

To conduct your assessment, you’ll need to address specific, measurable (or at least, recognizable) attributes of the interface. For example [Bro03]:

Intuitiveness. The degree to which the interface follows expected usage patterns so that even a novice can use it without significant training.

• Is the interface layout conducive to easy understanding?

• Are interface operations easy to locate and initiate?

• Does the interface use a recognizable metaphor?

• Is input specified to economize key strokes or mouse clicks?

• Does the interface follow the three golden rules? (Chapter 11)

• Do aesthetics aid in understanding and usage?

404 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

uote:

“Any activity becomes creative when the doer cares about doing it right, or better.”

John Updike

Although it’s tempting to develop quantitative measures for the quality factors noted here, you can also create a simple checklist of attributes that provide a solid indication that the factor is present.

3 These characteristics and questions would be addressed as part of a software review (Chapter 15).

pre75977_ch14.qxd 11/27/08 5:51 PM Page 404

Efficiency. The degree to which operations and information can be located or initiated.

• Does the interface layout and style allow a user to locate operations and information efficiently?

• Can a sequence of operations (or data input) be performed with an economy of motion?

• Are output data or content presented so that it is understood immediately?

• Have hierarchical operations been organized in a way that minimizes the depth to which a user must navigate to get something done?

Robustness. The degree to which the software handles bad input data or inap- propriate user interaction.

• Will the software recognize the error if data at or just outside prescribed boundaries is input? More importantly, will the software continue to operate without failure or degradation?

• Will the interface recognize common cognitive or manipulative mistakes and explicitly guide the user back on the right track?

• Does the interface provide useful diagnosis and guidance when an error condition (associated with software functionality) is uncovered?

Richness. The degree to which the interface provides a rich feature set.

• Can the interface be customized to the specific needs of a user?

• Does the interface provide a macro capability that enables a user to identify a sequence of common operations with a single action or command?

As the interface design is developed, the software team would review the design prototype and ask the questions noted. If the answer to most of these questions is “yes,” it is likely that the user interface exhibits high quality. A collection of questions similar to these would be developed for each quality factor to be assessed.

14.2.5 The Transition to a Quantitative View

In the preceding subsections, I have presented a variety of qualitative factors for the “measurement” of software quality. The software engineering community strives to develop precise measures for software quality and is sometimes frustrated by the subjective nature of the activity. Cavano and McCall [Cav78] discuss this situation:

The determination of quality is a key factor in every day events—wine tasting contests,

sporting events [e.g., gymnastics], talent contests, etc. In these situations, quality is

judged in the most fundamental and direct manner: side by side comparison of objects

under identical conditions and with predetermined concepts. The wine may be judged

according to clarity, color, bouquet, taste, etc. However, this type of judgment is very sub-

jective; to have any value at all, it must be made by an expert.

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 405

pre75977_ch14.qxd 11/27/08 5:51 PM Page 405

Subjectivity and specialization also apply to determining software quality. To help

solve this problem, a more precise definition of software quality is needed as well as a

way to derive quantitative measurements of software quality for objective analysis. . . .

Since there is no such thing as absolute knowledge, one should not expect to measure

software quality exactly, for every measurement is partially imperfect. Jacob Bronkowski

described this paradox of knowledge in this way: “Year by year we devise more precise

instruments with which to observe nature with more fineness. And when we look at the

observations we are discomfited to see that they are still fuzzy, and we feel that they are

as uncertain as ever.”

In Chapter 23, I’ll present a set of software metrics that can be applied to the quan- titative assessment of software quality. In all cases, the metrics represent indirect measures; that is, we never really measure quality but rather some manifestation of quality. The complicating factor is the precise relationship between the variable that is measured and the quality of software.

1 4 . 3 T H E S O F T WA R E Q U A L I T Y D I L E M M A In an interview [Ven03] published on the Web, Bertrand Meyer discusses what I call the quality dilemma:

If you produce a software system that has terrible quality, you lose because no one will

want to buy it. If on the other hand you spend infinite time, extremely large effort, and

huge sums of money to build the absolutely perfect piece of software, then it’s going to

take so long to complete and it will be so expensive to produce that you’ll be out of busi-

ness anyway. Either you missed the market window, or you simply exhausted all your

resources. So people in industry try to get to that magical middle ground where the prod-

uct is good enough not to be rejected right away, such as during evaluation, but also not

the object of so much perfectionism and so much work that it would take too long or cost

too much to complete.

It’s fine to state that software engineers should strive to produce high-quality systems. It’s even better to apply good practices in your attempt to do so. But the situation discussed by Meyer is real life and represents a dilemma for even the best software engineering organizations.

14.3.1 “Good Enough” Software

Stated bluntly, if we are to accept the argument made by Meyer, is it acceptable to produce “good enough” software? The answer to this question must be “yes,” because major software companies do it every day. They create software with known bugs and deliver it to a broad population of end users. They recognize that some of the functions and features delivered in Version 1.0 may not be of the highest quality and plan for improvements in Version 2.0. They do this knowing that some cus- tomers will complain, but they recognize that time-to-market may trump better qual- ity as long as the delivered product is “good enough.”

406 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

When you’re faced with the quality dilemma (and everyone is faced with it at one time or another), try to achieve balance— enough effort to produce acceptable quality without burying the project.

pre75977_ch14.qxd 11/27/08 5:51 PM Page 406

Exactly what is “good enough”? Good enough software delivers high-quality func- tions and features that users desire, but at the same time it delivers other more obscure or specialized functions and features that contain known bugs. The soft- ware vendor hopes that the vast majority of end users will overlook the bugs because they are so happy with other application functionality.

This idea may resonate with many readers. If you’re one of them, I can only ask you to consider some of the arguments against “good enough.”

It is true that “good enough” may work in some application domains and for a few major software companies. After all, if a company has a large marketing budget and can convince enough people to buy version 1.0, it has succeeded in locking them in. As I noted earlier, it can argue that it will improve quality in subsequent versions. By delivering a good enough version 1.0, it has cornered the market.

If you work for a small company be wary of this philosophy. When you deliver a good enough (buggy) product, you risk permanent damage to your company’s repu- tation. You may never get a chance to deliver version 2.0 because bad buzz may cause your sales to plummet and your company to fold.

If you work in certain application domains (e.g., real-time embedded software) or build application software that is integrated with hardware (e.g., automotive soft- ware, telecommunications software), delivering software with known bugs can be negligent and open your company to expensive litigation. In some cases, it can even be criminal. No one wants good enough aircraft avionics software!

So, proceed with caution if you believe that “good enough” is a short cut that can solve your software quality problems. It can work, but only for a few and only in a limited set of application domains.4

14.3.2 The Cost of Quality

The argument goes something like this—we know that quality is important, but it costs us time and money—too much time and money to get the level of software quality we really want. On its face, this argument seems reasonable (see Meyer’s comments ear- lier in this section). There is no question that quality has a cost, but lack of quality also has a cost—not only to end users who must live with buggy software, but also to the software organization that has built and must maintain it. The real question is this: which cost should we be worried about? To answer this question, you must understand both the cost of achieving quality and the cost of low-quality software.

The cost of quality includes all costs incurred in the pursuit of quality or in per- forming quality-related activities and the downstream costs of lack of quality. To understand these costs, an organization must collect metrics to provide a baseline for the current cost of quality, identify opportunities for reducing these costs, and provide a normalized basis of comparison. The cost of quality can be divided into costs associated with prevention, appraisal, and failure.

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 407

4 A worthwhile discussion of the pros and cons of “good enough” software can be found in [Bre02].

pre75977_ch14.qxd 11/27/08 5:52 PM Page 407

Prevention costs include (1) the cost of management activities required to plan and coordinate all quality control and quality assurance activities, (2) the cost of added technical activities to develop complete requirements and design models, (3) test planning costs, and (4) the cost of all training associated with these activities.

Appraisal costs include activities to gain insight into product condition the “first time through” each process. Examples of appraisal costs include:

• Cost of conducting technical reviews (Chapter 15) for software engineering work products

• Cost of data collection and metrics evaluation (Chapter 23)

• Cost of testing and debugging (Chapters 18 through 21)

Failure costs are those that would disappear if no errors appeared before or after shipping a product to customers. Failure costs may be subdivided into internal failure costs and external failure costs. Internal failure costs are incurred when you detect an error in a product prior to shipment. Internal failure costs include

• Cost required to perform rework (repair) to correct an error

• Cost that occurs when rework inadvertently generates side effects that must be mitigated

• Costs associated with the collection of quality metrics that allow an organi- zation to assess the modes of failure

External failure costs are associated with defects found after the product has been shipped to the customer. Examples of external failure costs are complaint resolution, product return and replacement, help line support, and labor costs associated with warranty work. A poor reputation and the resulting loss of business is another external failure cost that is difficult to quantify but nonetheless very real. Bad things happen when low-quality software is produced.

In an indictment of software developers who refuse to consider external failure costs, Cem Kaner [Kan95] states:

Many of the external failure costs, such as goodwill, are difficult to quantify, and many com-

panies therefore ignore them when calculating their cost-benefit tradeoffs. Other external

failure costs can be reduced (e.g. by providing cheaper, lower-quality, post-sale support, or

by charging customers for support) without increasing customer satisfaction. By ignoring

the costs to our customers of bad products, quality engineers encourage quality-related

decision-making that victimizes our customers, rather than delighting them.

As expected, the relative costs to find and repair an error or defect increase dra- matically as we go from prevention to detection to internal failure to external failure costs. Figure 14.2, based on data collected by Boehm and Basili [Boe01b] and illus- trated by Cigital Inc. [Cig07], illustrates this phenomenon.

The industry average cost to correct a defect during code generation is approxi- mately $977 per error. The industry average cost to correct the same error if it is

408 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Don’t be afraid to incur significant prevention costs. Rest assured that your investment will provide an excellent return.

uote:

“It takes less time to do a thing right than to explain why you did it wrong.”

H. W. Longfellow

pre75977_ch14.qxd 11/27/08 5:52 PM Page 408

discovered during system testing is $7,136 per error. Cigital Inc. [Cig07] considers a large application that has 200 errors introduced during coding.

According to industry average data, the cost of finding and correcting defects during the

coding phase is $977 per defect. Thus, the total cost for correcting the 200 “critical”

defects during this phase (200 ! $977) is approximately $195,400.

Industry average data shows that the cost of finding and correcting defects during the

system testing phase is $7,136 per defect. In this case, assuming that the system testing

phase revealed approximately 50 critical defects (or only 25% of those found by Cigital in

the coding phase), the cost of finding and fixing those defects (50 ! $7,136) would have

been approximately $356,800. This would also have resulted in 150 critical errors going

undetected and uncorrected. The cost of finding and fixing these remaining 150 defects

in the maintenance phase (150 ! $14,102) would have been $2,115,300. Thus, the total

cost of finding and fixing the 200 defects after the coding phase would have been

$2,472,100 ($2,115,300 " $356,800).

Even if your software organization has costs that are half of the industry average (most have no idea what their costs are!), the cost savings associated with early quality control and assurance activities (conducted during requirements analysis and design) are compelling.

14.3.3 Risks

In Chapter 1 of this book, I wrote “people bet their jobs, their comforts, their safety, their entertainment, their decisions, and their very lives on computer software. It better be right.” The implication is that low-quality software increases risks for both the devel- oper and the end user. In the preceding subsection, I discussed one of these risks (cost). But the downside of poorly designed and implemented applications does not always stop with dollars and time. An extreme example [Gag04] might serve to illustrate.

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 409

Requirements

$139 $455 $977

$7,136

$14,102

Design Coding Testing Maintenance

$16,000.00

$14,000.00

$12,000.00

$10,000.00

$8,000.00

$6,000.00

$4,000.00

$2,000.00

FIGURE 14.2 Relative cost of correcting errors and defects Source: Adapted from [Boe01b].

pre75977_ch14.qxd 11/27/08 5:52 PM Page 409

Throughout the month of November 2000 at a hospital in Panama, 28 patients received massive overdoses of gamma rays during treatment for a variety of cancers. In the months that followed, 5 of these patients died from radiation poisoning and 15 others developed serious complications. What caused this tragedy? A software package, developed by a U.S. company, was modified by hospital technicians to compute doses of radiation for each patient.

The three Panamanian medical physicists, who “tweeked” the software to provide additional capability, were charged with second-degree murder. The U.S. company is faced with serious litigation in two countries. Gage and McCormick comment:

This is not a cautionary tale for medical technicians, even though they can find them-

selves fighting to stay out of jail if they misunderstand or misuse technology. This also

is not a tale of how human beings can be injured or worse by poorly designed or poorly

explained software, although there are plenty of examples to make the point. This is a

warning for any creator of computer programs: that software quality matters, that appli-

cations must be foolproof, and that—whether embedded in the engine of a car, a robotic

arm in a factory or a healing device in a hospital—poorly deployed code can kill.

Poor quality leads to risks, some of them very serious.

14.3.4 Negligence and Liability

The story is all too common. A governmental or corporate entity hires a major soft- ware developer or consulting company to analyze requirements and then design and construct a software-based “system” to support some major activity. The system might support a major corporate function (e.g., pension management) or some gov- ernmental function (e.g., health care administration or homeland security).

Work begins with the best of intentions on both sides, but by the time the system is delivered, things have gone bad. The system is late, fails to deliver desired features and functions, is error-prone, and does not meet with customer approval. Litigation ensues.

In most cases, the customer claims that the developer has been negligent (in the manner in which it has applied software practices) and is therefore not entitled to payment. The developer often claims that the customer has repeatedly changed its requirements and has subverted the development partnership in other ways. In every case, the quality of the delivered system comes into question.

14.3.5 Quality and Security

As the criticality of Web-based systems and applications grows, application security has become increasingly important. Stated simply, software that does not exhibit high quality is easier to hack, and as a consequence, low-quality software can indi- rectly increase the security risk with all of its attendant costs and problems.

In an interview in ComputerWorld, author and security expert Gary McGraw com- ments [Wil05]:

Software security relates entirely and completely to quality. You must think about secu-

rity, reliability, availability, dependability—at the beginning, in the design, architecture,

test, and coding phases, all through the software life cycle [process]. Even people aware

410 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

pre75977_ch14.qxd 11/27/08 5:52 PM Page 410

of the software security problem have focused on late life-cycle stuff. The earlier you find

the software problem, the better. And there are two kinds of software problems. One is

bugs, which are implementation problems. The other is software flaws—architectural

problems in the design. People pay too much attention to bugs and not enough on flaws.

To build a secure system, you must focus on quality, and that focus must begin dur- ing design. The concepts and methods discussed in Part 2 of this book lead to a soft- ware architecture that reduces “flaws.” By eliminating architectural flaws (thereby improving software quality), you will make it far more difficult to hack the software.

14.3.6 The Impact of Management Actions

Software quality is often influenced as much by management decisions as it is by technology decisions. Even the best software engineering practices can be subverted by poor business decisions and questionable project management actions.

In Part 4 of this book I discuss project management within the context of the soft- ware process. As each project task is initiated, a project leader will make decisions that can have a significant impact on product quality.

Estimation decisions. As I note in Chapter 26, a software team is rarely given the luxury of providing an estimate for a project before delivery dates are established and an overall budget is specified. Instead, the team conducts a “sanity check” to ensure that delivery dates and milestones are rational. In many cases there is enormous time-to-market pressure that forces a team to accept unrealistic delivery dates. As a consequence, shortcuts are taken, activities that lead to higher-quality software may be skipped, and product quality suffers. If a delivery date is irrational, it is important to hold your ground. Explain why you need more time, or alternatively, suggest a subset of functionality that can be delivered (with high quality) in the time allotted.

Scheduling decisions. When a software project schedule is established (Chapter 27), tasks are sequenced based on dependencies. For example, because component A depends on processing that occurs within components B, C, and D, component A cannot be scheduled for testing until components B, C, and D are fully tested. A project schedule would reflect this. But if time is very short, and A must be available for further critical testing, you might decide to test A without its subordi- nate components (which are running slightly behind schedule), so that you can make it available for other testing that must be done before delivery. After all, the deadline looms. As a consequence, A may have defects that are hidden, only to be discovered much later. Quality suffers.

Risk-oriented decisions. Risk management (Chapter 28) is one of the key attrib- utes of a successful software project. You really do need to know what might go wrong and establish a contingency plan if it does. Too many software teams prefer blind optimism, establishing a development schedule under the assumption that nothing will go wrong. Worse, they don’t have a way of handling things that do go wrong. As a consequence, when a risk becomes a reality, chaos reigns, and as the degree of craziness rises, the level of quality invariably falls.

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 411

pre75977_ch14.qxd 11/27/08 5:52 PM Page 411

The software quality dilemma can best be summarized by stating Meskimen’s Law—There’s never time to do it right, but always time to do it over again. My advice: taking the time to do it right is almost never the wrong decision.

1 4 . 4 A C H I E V I N G S O F T WA R E Q U A L I T Y Software quality doesn’t just appear. It is the result of good project management and solid software engineering practice. Management and practice are applied within the context of four broad activities that help a software team achieve high software quality: software engineering methods, project management techniques, quality control actions, and software quality assurance.

14.4.1 Software Engineering Methods

If you expect to build high-quality software, you must understand the problem to be solved. You must also be capable of creating a design that conforms to the problem while at the same time exhibiting characteristics that lead to software that exhibits the quality dimensions and factors discussed in Section 14.2.

In Part 2 of this book, I presented a wide array of concepts and methods that can lead to a reasonably complete understanding of the problem and a comprehensive design that establishes a solid foundation for the construction activity. If you apply those concepts and adopt appropriate analysis and design methods, the likelihood of creating high-quality software will increase substantially.

14.4.2 Project Management Techniques

The impact of poor management decisions on software quality has been discussed in Section 14.3.6. The implications are clear: if (1) a project manager uses estimation to verify that delivery dates are achievable, (2) schedule dependencies are understood and the team resists the temptation to use short cuts, (3) risk planning is conducted so problems do not breed chaos, software quality will be affected in a positive way.

In addition, the project plan should include explicit techniques for quality and change management. Techniques that lead to good project management practices are discussed in Part 4 of this book.

14.4.3 Quality Control

Quality control encompasses a set of software engineering actions that help to ensure that each work product meets its quality goals. Models are reviewed to ensure that they are complete and consistent. Code may be inspected in order to uncover and correct errors before testing commences. A series of testing steps is applied to uncover errors in processing logic, data manipulation, and interface communication. A combination of measurement and feedback allows a software team to tune the process when any of these work products fail to meet quality goals. Quality control activities are discussed in detail throughout the remainder of Part 3 of this book.

412 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

What do I need to do to

affect quality in a positive way?

What is software

quality control? ?

pre75977_ch14.qxd 11/27/08 5:52 PM Page 412

14.4.4 Quality Assurance

Quality assurance establishes the infrastructure that supports solid software engi- neering methods, rational project management, and quality control actions—all pivotal if you intend to build high-quality software. In addition, quality assurance consists of a set of auditing and reporting functions that assess the effectiveness and completeness of quality control actions. The goal of quality assurance is to provide management and technical staff with the data necessary to be informed about prod- uct quality, thereby gaining insight and confidence that actions to achieve product quality are working. Of course, if the data provided through quality assurance iden- tifies problems, it is management’s responsibility to address the problems and apply the necessary resources to resolve quality issues. Software quality assurance is dis- cussed in detail in Chapter 16.

1 4 . 5 S U M M A R Y Concern for the quality of the software-based systems has grown as software becomes integrated into every aspect of our daily lives. But it is difficult to develop a comprehensive description of software quality. In this chapter quality has been defined as an effective software process applied in a manner that creates a useful product that provides measurable value for those who produce it and those who use it.

A wide variety of software quality dimensions and factors have been proposed over the years. All try to define a set of characteristics that, if achieved, will lead to high software quality. McCall’s and the ISO 9126 quality factors establish character- istics such as reliability, usability, maintainability, functionality, and portability as indicators that quality exists.

Every software organization is faced with the software quality dilemma. In essence, everyone wants to build high-quality systems, but the time and effort required to produce “perfect” software are simply unavailable in a market-driven world. The question becomes, should we build software that is “good enough”? Although many companies do just that, there is a significant downside that must be considered.

Regardless of the approach that is chosen, quality does have a cost that can be discussed in terms of prevention, appraisal, and failure. Prevention costs include all software engineering actions that are designed to prevent defects in the first place. Appraisal costs are associated with those actions that assess software work prod- ucts to determine their quality. Failure costs encompass the internal price of failure and the external effects that poor quality precipitates.

Software quality is achieved through the application of software engineering methods, solid management practices, and comprehensive quality control—all sup- ported by a software quality assurance infrastructure. In the chapters that follow, quality control and assurance are discussed in some detail.

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 413

WebRef Useful links to SQA resources can be found at www.niwotridge .com/Resources/ PM- SWEResources/ SoftwareQuality Assurance.htm

pre75977_ch14.qxd 11/27/08 5:52 PM Page 413

P R O B L E M S A N D P O I N T S T O P O N D E R 14.1. Describe how you would assess the quality of a university before applying to it. What factors would be important? Which would be critical?

14.2. Garvin [Gar84] describes five different views of quality. Provide an example of each using one or more well-known electronic products with which you are familiar.

14.3. Using the definition of software quality proposed in Section 14.2, do you think it’s possi- ble to create a useful product that provides measurable value without using an effective process? Explain your answer.

14.4. Add two additional questions to each of Garvin’s quality dimensions presented in Section 14.2.1.

14.5. McCall’s quality factors were developed during the 1970s. Almost every aspect of computing has changed dramatically since the time that they were developed, and yet, McCall’s factors continue to apply to modern software. Can you draw any conclusions based on this fact?

14.6. Using the subattributes noted for the ISO 9126 quality factor “maintainability” in Section 14.2.3, develop a set of questions that explore whether or not these attributes are present. Follow the example shown in Section 14.2.4.

14.7. Describe the software quality dilemma in your own words.

14.8. What is “good enough” software? Name a specific company and specific products that you believe were developed using the good enough philosophy.

14.9. Considering each of the four aspects of the cost of quality, which do you think is the most expensive and why?

14.10. Do a Web search and find three other examples of “risks” to the public that can be directly traced to poor software quality. Consider beginning your search at http:// catless.ncl.ac.uk/risks.

14.11. Are quality and security the same thing? Explain.

14.12. Explain why it is that many of us continue to live by Meskimen’s law. What is it about the software business that causes this?

F U R T H E R R E A D I N G S A N D I N F O R M AT I O N S O U R C E S Basic software quality concepts are considered in books by Henry and Hanlon (Software Quality Assurance, Prentice-Hall, 2008), Khan and his colleagues (Software Quality: Concepts and Practice, Alpha Science International, Ltd., 2006), O’Regan (A Practical Approach to Software Quality, Springer, 2002), and Daughtrey (Fundamental Concepts for the Software Quality Engineer, ASQ Quality Press, 2001).

Duvall and his colleagues (Continuous Integration: Improving Software Quality and Reducing Risk, Addison-Wesley, 2007), Tian (Software Quality Engineering, Wiley-IEEE Computer Society Press, 2005), Kandt (Software Engineering Quality Practices, Auerbach, 2005), Godbole (Software Quality Assurance: Principles and Practice, Alpha Science International, Ltd., 2004), and Galin (Software Quality Assurance: From Theory to Implementation, Addison-Wesley, 2003) present detailed treatments of SQA. Quality assurance in the context of the agile process is considered by Stamelos and Sfetsos (Agile Software Development Quality Assurance, IGI Global, 2007).

Solid design leads to high software quality. Jayasawal and Patton (Design for Trustworthy Software, Prentice-Hall, 2006) and Ploesch (Contracts, Scenarios and Prototypes, Springer, 2004) discuss tools and techniques for developing “robust” software.

414 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

pre75977_ch14.qxd 11/27/08 5:52 PM Page 414

Measurement is an important component of software quality engineering. Ejiogu (Software Metrics: The Discipline of Software Quality, BookSurge Publishing, 2005), Kan (Metrics and Mod- els in Software Quality Engineering, Addison-Wesley, 2002), and Nance and Arthur (Managing Software Quality, Springer, 2002) discuss important quality-related metrics and models. The team-oriented aspects of software quality are considered by Evans (Achieving Software Quality through Teamwork, Artech House Publishers, 2004).

A wide variety of information sources on software quality is available on the Internet. An up-to- date list of World Wide Web references relevant to software quality can be found at the SEPA website: www.mhhe.com/engcs/compsci/pressman/professional/olc/ser.htm.

C H A P T E R 1 4 Q U A L I T Y C O N C E P T S 415

pre75977_ch14.qxd 11/27/08 5:52 PM Page 415

Cover Page
Title Page
Copyright Page
Dedication
About Author Page
Preface
CONTENTS AT A GLANCE
CONTENTS
CHAPTER 1: SOFTWARE AND SOFTWARE ENGINEERING

1.1 The Nature of Software

1.1.1 Defining Software
1.1.2 Software Application Domains
1.1.3 Legacy Software

1.2 The Unique Nature of WebApps
1.3 Software Engineering
1.4 The Software Process
1.5 Software Engineering Practice

1.5.1 The Essence of Practice
1.5.2 General Principles

1.6 Software Myths
1.7 How It All Starts
1.8 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

PART ONE: THE SOFTWARE PROCESS

CHAPTER 2: PROCESS MODELS

2.1 A Generic Process Model
2.2 Process Assessment and Improvement
2.3 Prescriptive Process Models
2.4 Specialized Process Models
2.5 The Unified Process
2.6 Personal and Team Process Models
2.7 Process Technology
2.8 Product and Process
2.9 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 3: AGILE DEVELOPMENT

3.1 What Is Agility?
3.2 Agility and the Cost of Change
3.3 What Is an Agile Process?
3.4 Extreme Programming (XP)
3.5 Other Agile Process Models
3.6 A Tool Set for the Agile Process
3.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

PART TWO: MODELING

CHAPTER 4: PRINCIPLES THAT GUIDE PRACTICE

4.1 Software Engineering Knowledge
4.2 Core Principles
4.3 Principles That Guide Each Framework Activity
4.4 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 5: UNDERSTANDING REQUIREMENTS

5.1 Requirements Engineering
5.2 Establishing the Groundwork
5.3 Eliciting Requirements
5.4 Developing Use Cases
5.5 Building the Requirements Model
5.6 Negotiating Requirements
5.7 Validating Requirements
5.8 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 6: REQUIREMENTS MODELING: SCENARIOS, INFORMATION, AND ANALYSIS CLASSES

6.1 Requirements Analysis
6.2 Scenario-Based Modeling
6.3 UML Models That Supplement the Use Case
6.4 Data Modeling Concepts
6.5 Class-Based Modeling
6.6 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 7: REQUIREMENTS MODELING: FLOW, BEHAVIOR, PATTERNS, AND WEBAPPS

7.1 Requirements Modeling Strategies
7.2 Flow-Oriented Modeling
7.3 Creating a Behavioral Model
7.4 Patterns for Requirements Modeling
7.5 Requirements Modeling for WebApps
7.6 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 8: DESIGN CONCEPTS

8.1 Design within the Context of Software Engineering
8.2 The Design Process
8.3 Design Concepts
8.4 The Design Model
8.5 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 9: ARCHITECTURAL DESIGN

9.1 Software Architecture
9.2 Architectural Genres
9.3 Architectural Styles
9.4 Architectural Design
9.5 Assessing Alternative Architectural Designs
9.6 Architectural Mapping Using Data Flow
9.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 10: COMPONENT-LEVEL DESIGN

10.1 What Is a Component?
10.2 Designing Class-Based Components
10.3 Conducting Component-Level Design
10.4 Component-Level Design for WebApps
10.5 Designing Traditional Components
10.6 Component-Based Development
10.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 11: USER INTERFACE DESIGN

11.1 The Golden Rules
11.2 User Interface Analysis and Design
11.3 Interface Analysis
11.4 Interface Design Steps
11.5 WebApp Interface Design
11.6 Design Evaluation
11.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 12: PATTERN-BASED DESIGN

12.1 Design Patterns
12.2 Pattern-Based Software Design
12.3 Architectural Patterns
12.4 Component-Level Design Patterns
12.5 User Interface Design Patterns
12.6 WebApp Design Patterns
12.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READING AND INFORMATION SOURCES

CHAPTER 13: WEBAPP DESIGN

13.1 WebApp Design Quality
13.2 Design Goals
13.3 A Design Pyramid for WebApps
13.4 WebApp Interface Design
13.5 Aesthetic Design
13.6 Content Design
13.7 Architecture Design
13.8 Navigation Design
13.9 Component-Level Design
13.10 Object-Oriented Hypermedia Design Method (OOHDM
13.11 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

PART THREE: QUALITY MANAGEMENT

CHAPTER 14: QUALITY CONCEPTS

14.1 What Is Quality?
14.2 Software Quality
14.3 The Software Quality Dilemma
14.4 Achieving Software Quality
14.5 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 15: REVIEW TECHNIQUES

15.1 Cost Impact of Software Defects
15.2 Defect Amplification and Removal
15.3 Review Metrics and Their Use
15.4 Reviews: A Formality Spectrum
15.5 Informal Reviews
15.6 Formal Technical Reviews
15.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 16: SOFTWARE QUALITY ASSURANCE

16.1 Background Issues
16.2 Elements of Software Quality Assurance
16.3 SQA Tasks, Goals, and Metrics
16.4 Formal Approaches to SQA
16.5 Statistical Software Quality Assurance
16.6 Software Reliability
16.7 The ISO 9000 Quality Standards
16.8 The SQA Plan
16.9 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 17: SOFTWARE TESTING STRATEGIES

17.1 A Strategic Approach to Software Testing
17.2 Strategic Issues
17.3 Test Strategies for Conventional Software
17.4 Test Strategies for Object-Oriented Software
17.5 Test Strategies for WebApps
17.6 Validation Testing
17.7 System Testing
17.8 The Art of Debugging
17.9 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 18: TESTING CONVENTIONAL APPLICATIONS

18.1 Software Testing Fundamentals
18.2 Internal and External Views of Testing
18.3 White-Box Testing
18.4 Basis Path Testing
18.5 Control Structure Testing
18.6 Black-Box Testing
18.7 Model-Based Testing
18.8 Testing for Specialized Environments, Architectures, and Applications
18.9 Patterns for Software Testing
18.10 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 19: TESTING OBJECT-ORIENTED APPLICATIONS

19.1 Broadening the View of Testing
19.2 Testing OOA and OOD Models
19.3 Object-Oriented Testing Strategies
19.4 Object-Oriented Testing Methods
19.5 Testing Methods Applicable at the Class Level
19.6 Interclass Test-Case Design
19.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 20: TESTING WEB APPLICATIONS

20.1 Testing Concepts for WebApps
20.2 The Testing Process—An Overview
20.3 Content Testing
20.4 User Interface Testing
20.5 Component-Level Testing
20.6 Navigation Testing
20.7 Configuration Testing
20.8 Security Testing
20.9 Performance Testing
20.10 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 21: FORMAL MODELING AND VERIFICATION

21.1 The Cleanroom Strategy
21.2 Functional Specification
21.3 Cleanroom Design
21.4 Cleanroom Testing
21.5 Formal Methods Concepts
21.6 Applying Mathematical Notation for Formal Specification
21.7 Formal Specification Languages
21.8 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 22: SOFTWARE CONFIGURATION MANAGEMENT

22.1 Software Configuration Management
22.2 The SCM Repository
22.3 The SCM Process
22.4 Configuration Management for WebApps
22.5 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 23: PRODUCT METRICS

23.1 A Framework for Product Metrics
23.2 Metrics for the Requirements Model
23.3 Metrics for the Design Model
23.4 Design Metrics for WebApps
23.5 Metrics for Source Code
23.6 Metrics for Testing
23.7 Metrics for Maintenance
23.8 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

PART FOUR: MANAGING SOFTWARE PROJECTS

CHAPTER 24: PROJECT MANAGEMENT CONCEPTS

24.1 The Management Spectrum
24.2 People
24.3 The Product
24.4 The Process
24.5 The Project
24.6 The W5HH Principle
24.7 Critical Practices
24.8 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 25: PROCESS AND PROJECT METRICS

25.1 Metrics in the Process and Project Domains
25.2 Software Measurement
25.3 Metrics for Software Quality
25.4 Integrating Metrics within the Software Process
25.5 Metrics for Small Organizations
25.6 Establishing a Software Metrics Program
25.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 26: ESTIMATION FOR SOFTWARE PROJECTS

26.1 Observations on Estimation
26.2 The Project Planning Process
26.3 Software Scope and Feasibility
26.4 Resources
26.5 Software Project Estimation
26.6 Decomposition Techniques
26.7 Empirical Estimation Models
26.8 Estimation for Object-Oriented Projects
26.9 Specialized Estimation Techniques
26.10 The Make/Buy Decision
26.11 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 27: PROJECT SCHEDULING

27.1 Basic Concepts
27.2 Project Scheduling
27.3 Defining a Task Set for the Software Project
27.4 Defining a Task Network
27.5 Scheduling
27.6 Earned Value Analysis
27.7 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 28: RISK MANAGEMENT

28.1 Reactive versus Proactive Risk Strategies
28.2 Software Risks
28.3 Risk Identification
28.4 Risk Projection
28.5 Risk Refinement
28.6 Risk Mitigation, Monitoring, and Management
28.7 The RMMM Plan
28.8 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 29: MAINTENANCE AND REENGINEERING

29.1 Software Maintenance
29.2 Software Supportability
29.3 Reengineering
29.4 Business Process Reengineering
29.5 Software Reengineering
29.6 Reverse Engineering
29.7 Restructuring
29.8 Forward Engineering
29.9 The Economics of Reengineering
29.10 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

PART FIVE: ADVANCED TOPICS

CHAPTER 30: SOFTWARE PROCESS IMPROVEMENT

30.1 What Is SPI?
30.2 The SPI Process
30.3 The CMMI
30.4 The People CMM
30.5 Other SPI Frameworks
30.6 SPI Return on Investment
30.7 SPI Trends
30.8 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 31: EMERGING TRENDS IN SOFTWARE ENGINEERING

31.1 Technology Evolution
31.2 Observing Software Engineering Trends
31.3 Identifying “Soft Trends”
31.4 Technology Directions
31.5 Tools-Related Trends
31.6 Summary
PROBLEMS AND POINTS TO PONDER
FURTHER READINGS AND INFORMATION SOURCES

CHAPTER 32: CONCLUDING COMMENTS

32.1 The Importance of Software—Revisited
32.2 People and the Way They Build Systems
32.3 New Modes for Representing Information
32.4 The Long View
32.5 The Software Engineer’s Responsibility
32.6 A Final Comment

APPENDIX 1: AN INTRODUCTION TO UML
APPENDIX 2: OBJECT-ORIENTED CONCEPTS
REFERENCES
INDEX

Mitigating Quality Risks Overview This assignment is intended to solidify your understanding of how and when some of the techniques we are learning about can address specific software quality risks. You will compare and contrast how scenario-based usability engineering and architecture-oriented design affect different quality attributes. The background for this assignment will come from prior reading assignments and one new resource. The prior readings that are most relevant to this assignment include:

 Pressman's Chapter 14  Chapter 3 in Scenario-Based Usability Engineering The new resource is: Air Traffic Control--A Case Study in Designing for High Availability

Assignment Both Chapter 3 from Scenario-Based Usability Engineering and the case study from Bass' book focus on the design of air traffic control systems. The first focuses on the use of usability engineering techniques, including participatory design and scenario-based design, as a requirements gathering technique, explaining how that played out in an air traffic control project. The case study from Bass instead focuses on the software architecture that was used in an air traffic control system as an example of how software architecture choices can be used to address challenges in complex systems.

The main task of your assignment is to evaluate both scenario-based usability engineering and software architecture-based design in terms of how they affect each software quality attribute, using McCall's list of software quality attributes (see 14.2.2 in Pressman's Chapter 14). Systematically go through each of the quality attributes in that list and consider whether (and how) both scenario-based usability engineering and architecture-based design addressed or helped achieve that specific quality attribute.

Write up your results in a short paper (about 4-5 pages) that you will turn in. Be sure to include the following elements in your solution:

1. Summary: provide a brief description of the two methods being compared, using your own words, and provide an introduction to the remainder of your paper.

2. Comparison: Using a separate subsection for each of the quality attributes listed in McCall's list, briefly but explicitly describe how scenario-based usability engineering affects that attribute (or whether it has no significant effect). You can use (but are not limited to) the air traffic control examples as source material to ground your argument. Your goal is to conclude whether scenario- based usability engineering will help significantly increase the chances of meeting a project's goal with respect to that quality attribute. Identify what you believe are the important characteristics/properties of a software project that make scenario- based usability engineering an important consideration for that specific quality attribute on the project.

Then perform the same analysis for architecture-driven design with respect to that quality (i.e., handle both techniques for one quality attribute in each subsection, with a separate subsection for each quality attribute in McCall's list).

3. Conclusions: Look back over the analysis you have performed, and summarize which group of quality attributes are affected by scenario-based usability engineering, and what characteristics of a software project make scenario-based usability engineering relevant in order to meet desired quality goals. Similarly, summarize which group of quality attributes are affected by architecture-driven design, and what characteristics of a software project make architectural decisions relevant in order to meet desired quality goals.

Assessment The following rubric will be used to assess your work:

Criteria Points

Project Summary 10 points

Excellent: Provides a clear explanation of both techniques and gives an appropriate overview of the comparative analysis you are performing, including a summary of the quality attributes used as the basis for the comparison.

10/10

Criteria Points

Good: Provides an explanation of both techniques and lists the quality attributes used in the analysis, but there is clear room for improving the overview/introduction so it is more understandable.

8/10

Satisfactory: Names the techniques to be compared, but does not provide a clear explanation of at least one of the techniques, or fails to clearly articulate the quality attributes used in the analysis.

6/10

Poor: Provides an introductory summary that does not clearly decribe either of the techniques analyzed.

3/10

No attempt: Section is missing. 0/10

Comparison 45 points

Excellent: Provides a clear, specific evaluation of how each of the two techniques helps support (or does not support) each of the quality attributes in the required list. The analysis is clearly broken into sections based on the quality attribute list, and both techniques are directly address in every section. Identifies the properties of a software system that make each technique appropriate, when the technique can directly affect software quality.

45/45

Good: Provides a clear evaluation of the two techniques for each quality attribute, with the analysis broken down into sections based on the quality attribute list indicated. A few sections may not provide a clear analysis of one of the techniques, or may fail to describe the properties of a software system that make the technique relevant.

36/45

Satisfactory: All quality attributes are addressed, but many sections do not clearly describe the effects of both techniques on the attribute, mischaracterize the effects, or fail to clearly identify the properties of software systems that make the technique relevant.

27/45

Poor: Simply restates basic concepts from the reading without contributing a useful discussion of when/why a technique addresses each quality attribute.

18/45

Criteria Points

No attempt: Section is missing. 0/45

Conclusions 35 points

Excellent: Explicitly identifies which quality attributes are significantly affected by each of the two techniques analyzed, what characteristics of a software project make the technique important to use, and generalizes from this a coherent description of "when" each of the two techniques should be applied (or at least considered). Also discusses what kind of project would be appropriate for applying both techniques.

35/35

Good: Identifies which quality attributes are significantly affected by each of the two techniques analyzed, and what characteristics of a software project make the technique important to use..

28/35

Satisfactory: Identifies which quality attributes are significantly affected by each of the two techniques analyzed. One of more of the attributes may be glossed over or omitted. Some important system characteristics necessary for a technique to be important may be identified, but some may be missing.

22/35

Poor: Attempts to evaluate the four architectures, but without any clear connection to the presented evaluation critieria or any clear summary of strengths and weaknesses for each architecture.

16/35

No attempt: Section is missing. 0/10

Writing/Presentation 10 points

Excellent: All writing is clear and readable, without grammar errors, punctuation errors, or other writing problems. Figures or summary tables are used appropriately where they clarify presentation. Clear headings and a cohesive document organization are used. The whole document looks like a professionally prepared report.

10/10

Criteria Points

Good: All writing is clear and readable, with only occasional minor writing errors. An appropriate attempt is made to use appropriate figures or summary tables where they help clarify presentation. Clear headings and a cohesive document organization are used.

8/10

Satisfactory: The document is readable, but some significant errors appear in the writing and/or organization. Some parts of the exposition may not communicate clearly. Summary representations of key portions of the work may be missing. Clear headings are used.

6/10

Poor: The document is readable, but includes significant errors in both writing and organization that make portions of the document hard to follow.

3/10

No attempt: Section is missing. 0/10

Total 100 points

Mitigating Quality Risks

Overview
 Pressman's Chapter 14
 Chapter 3 in Scenario-Based Usability Engineering
The new resource is:
Air Traffic Control--A Case Study in Designing for High Availability
Assignment
Assessment

blog1

Get help from top-rated tutors in any subject.