John M. Carroll
IBM Watson Research Center
Box 704
Yorktown Heights, New York 10598
This article originally appeared in Communications of the ACM, December 1994, Volume 37, Number 12, pp. 29-35. It is provided here with the permission of the author, who can now be contacted at the Department of Computer Science, Virginia Tech, Blacksburg, VA 24061.
Computers are more than just narrow functionality. They unavoidably restructure human activities, creating new possibilities as well as new difficulties. Conversely, each context in which humans experience and act provides detailed constraint for the development and application of computer technologies. In analyzing and designing systems and software we need better means to talk about how they may transform and/or be constrained by the contexts of user activity: this is the only way we can hope to attain control over the design of useful and usable systems. A direct approach is to explicitly envision and document typical and significant user activities early and continuingly in the development process. Such descriptions, often called "scenarios," support reasoning about situations of use, even before those situations are actually created.
In this paper, I survey and illustrate a method in which scenarios provide an integration of task analysis and design envisionment--a basic level of design representation. Scenario representations can be elaborated as prototypes, through the use of storyboard, video, and rapid prototyping tools. They are the minimal contexts for developing user-oriented design rationale: a given design decision can be evaluated and documented in terms of its specific consequences within particular scenarios. Scenarios and the elements of scenario-based design rationale can be generalized and abstracted using theories of human activity, enabling the cumulation and development of knowledge attained in the course of design. Thus, scenarios can provide a framework for a design-based science of human-computer interaction.
The real use of systems can be concretely described by the set of user interaction scenarios obtained when people try to use a system. Box 1 presents an observed example of an interaction scenario. An experienced procedural programmer is using a Smalltalk/V PM tutorial, working on a exercise to implement a new kind of text pane that displays its input in uppercase. This project focuses on concrete, albeit somewhat arcane, functionality; it involves learning about two key subtrees of the Smalltalk class hierarchy (Magnitude-Character and Window-SubPane-TextPane) and allows the programmer to implement a concrete and usable new Smalltalk object by extending existing system functionality with a minimum of coding. The programmer carries out the project by subclassing the TextPane class and specializing the code that processes character input from the keyboard. The scenario is a vivid and succinct description of a person learning something about Smalltalk; a comprehensive set of such scenarios would be a very concrete task analysis of the Smalltalk tutorial situation.
Even this single scenario is an obvious and immediate design resource: It highlights goals suggested by the appearance and behavior of the system, what people try to do with the system, what procedures are adopted, not adopted, carried out successfully or erroneously, and what interpretations people make of what happens to them. One can use it to identify problem areas for further design work. For example, in Box 1, the programmer has some difficulty navigating the Smalltalk class hierarchy--the tutorial could be improved by incorporating an indexing tool. A more subtle example, but an important one in the Smalltalk domain, is that the programmer in Box 1 copied a method to a subclass instead of inheriting the functionality and merely implementing the specialization in the subclass.
Projecting scenarios in design. We cannot rely only on observation to build a scenario representation. If we want scenarios to play a proactive role in guiding design and development, we need to be able to construct scenario representations for systems before any version is deployed in the field (Carroll & Rosson, 1990). In this case we are representing anticipated use. We can do this in part by analogy: the new system may be similar to other contemporary systems or "descended" from earlier systems--the characteristic scenarios for these related existing systems can be observed and analyzed.
We can also make use of theories of human activity (Norman, 1986); that is, we can try to instantiate known categories of human activity in the envisioned context of the system we are developing. For example, people must orient to novel task situations, identifying and analyzing appropriate goals; later, they must evaluate their own actions to determine whether those goals were attained (e.g., Carroll & Rosson, 1990; Norman, 1986). In building a scenario representation for a system in development, we can envision possible goal orientation and evaluation situations. As we develop such scenarios, we can track specific impacts of our design decisions for these theoretical categories of human activity.
The novelty of this proposal resides simply in the suggestion that scenarios be systematized as working design representations, for it is well-known that scenarios are spontaneously and pervasively used in system development as practiced now. Studies of actual design situations document the use of fairly detailed conjectures about what users will do with the system (e.g., Carroll, Thomas and Malhotra, 1979; Lammers, 1986; Rosson, Maass and Kellogg, 1987).
It has also been shown that users and designers can communicate effectively about system requirements and design options through narrative descriptions of envisioned use (Ehn, 1988; Greenbaum & Kyng, 1991). Indeed, the current state of practice more or less requires a substantial set of scenarios both for usability testing (Roberts & Moran, 1983) and for documentation and training design (Carroll, 1990). User interaction scenarios are typical in contemporary functional specification documents (officially they are included as illustrations of the specification, but they often appear before any of the specification, and developers intimate that they are sometimes used in lieu of the specification). In sum, scenarios are already everywhere in the design process. Systematically developing and managing a set of scenarios as part of the working design representation can help integrate and leverage these diverse, informal, and currently uncoordinated uses. Indeed, the costs of doing more comprehensive scenario development might actually be less than the distributed costs of current practice.
Our experience. We have used scenario-based design representations in a variety of our recent projects designing tools and environments for Smalltalk instruction and programming (Carroll & Rosson, 1991; Carroll & Rosson, 1992; Carroll, Singley & Rosson, 1992; Rosson & Carroll, 1993; Singley & Carroll, 1995; Singley, Carroll & Alpert, 1991). Our experience is therefore tilted toward open-ended, construction domains, though one might expect that scenario-based methods would apply at least as well in relatively more circumscribed application domains (in which the user tasks are often literally enumerable). We have found that fewer than a dozen scenarios provides a useful and manageable design representation.
Box 2 describes one of the scenarios envisioned to drive the design of the Smalltalk Guru tool in our MoleHill intelligent tutoring system (Alpert, Singley & Carroll, 1995; Singley & Carroll, 1995; Singley et al., 1991). The Guru tool provides guidance to the inclination people typically have for evaluating their own activity: just as the programmer finishes the project, the Guru offers suggestions about how to achieve a more modular code design, in this scenario exploiting Smalltalk's inheritance mechanism via the special variable "super". This example shows how empirical analysis of interaction scenarios for existing situations (for example, the Smalltalk tutorial situation described in Box 1) and abstractions from theories of human activity (the notion of an evaluation scenario) can provide proactive guidance in developing a scenario representation for a new system.
In our work, scenarios are more or less continually developed and reinterpreted. The scenario in Box 2 is a development of the scenario in Box 1, but it is not a final representation. It is merely a use-oriented view of the current design, a working representation. Different scenarios can conflict and "compete" in such a working design representation, forcing out greater clarity in the developing vision of the system.
Developing scenarios through the coordinated use of direct empirical observation and abstractions from theories of human activity has the additional benefit of continually emphasizing to the designer the continuities among various design projects. System design is of course highly creative, but few systems are completely novel. Indeed, the history of technology is a history of cumulative development and evolution (Basalla, 1984). Nevertheless, it is easy in the context of design work to focus singularly on the project at hand and lose the benefit of whatever might have been done and learned before. Our deliberate development of the Guru scenario helped us to appreciate that the Guru was a sort of bug critic (e.g., Brown, Burton & de Kleer, 1982) and thus is systematically related to other tutorial situations that had been designed and studied before.
Scenario-based design representations raise many challenges of their own. It is clear to anyone how to build an alphabetical list of functions. Building a set of user interaction narratives may be more appropriate, but it is also more ill-defined. Traditional functional specifications are frequently inconsistent or incomplete with respect to the user's needs and predilections (this was one of the cataclysmic discoveries of the 1980s), but because they are organized function by function, they are at least easy to align with the detailed design and implementation of the system. Two important current frontiers for scenario-based design methodology are (1) heuristics to help ensure broad sampling of scenarios in building a representation, and (2) tools for verifying and managing the alignment of scenario representations with detailed designs and implementations (Carey, McKerlie & Wilson, 1995; Carroll, 1995; Rosson & Carroll, 1995).
Explaining scenarios. It can be valuable to make explicit a causal account or rationale for a scenario. Indeed, envisioning a "better" scenario for users may well require making explicit just what it was that needed to be improved in terms of specific consequences for users, how improvements were attempted in terms of managing or altering consequences, and what other consequences might have been entrained as side-effects. A direct approach to articulating why scenarios run as they do is to isolate individual causal relationships between features of a situation of use and specific consequences for users:
IN [situation], [a feature] CAUSES ["desirable" psychological consequence]
BUT MAY ALSO CAUSE ["undesirable" psychological consequence]
Box 3 presents four causal relationships implicit in the UpperCasePane scenario (of Box 1). Thus, the Code Library Relation (a) asserts that the class hierarchy encourages standard design decompositions and reuse-oriented programming strategies. These are desirable consequences for the programmer working on the tutorial project. A downside, or undesirable consequence of the large code library is that Smalltalk programmers may experience problems searching for code to reuse or making appropriate use of code. The Names Relation (b) observes that while descriptive class and method names may help programmers quickly identify relevant functionality, the names may also mislead. The Navigation Relation (c) is a hypothesis that searching the class library may cause the programmer to incidentally learn something about the functionality in the library, though the sheer size of the library may make it difficult to learn much, particularly if the programmer is frustrated in his or her search. Finally, the Programming Tools Relation (d) observes that the tool-rich, interpreted Smalltalk environment may support a very fluid and opportunistic style of code development; the downside is that programmers may not reflect enough on the designs they are generating.
A psychological design rationale such as this can be developed by asking questions to elaborate the events in a use scenario (Carroll & Rosson, 1992). Thus, for the UpperCasePane scenario (Box 1), we might ask: Why did the programmer search for relevant functionality while still defining a strategy toward the project? Did anything about the situation make it especially difficult or easy to proceed in this way? What was the programmer thinking about while browsing for relevant classes and methods? Did the programmer have any particular difficulties with browsing; did he or she benefit from the browsing activity? Why did the programmer pursue several incorrect possibilities when searching for relevant functionality? Why was appendText: considered, and why was it finally rejected? Why did the programmer have trouble finding, and then later lose track of, Character class? Why did the programmer copy down the method code for characterInput:, instead of inheriting the functionality using "super"? How did the programmer decide to use Character's asUpperCase method? Why does the programmer test only one UpperCasePane, and test only the external behavior of the pane?
The answers to such questions are a set of hypotheses about the underlying causal relations in the scenario, the analyst's hypotheses about why and how an observed (or envisioned) scenario of use could occur. The objective of raising these questions and articulating hypothesized causal relations is to open up a process of critique and refinement, to help designers interrogate and explain key scenarios in enough detail that they can be interpreted, revised, perhaps discarded as the design proceeds. It is important to emphasize that a psychological design rationale, such as the fragment in Box 3, does not necessarily equate to the personal intentions of the designers. We know that the designers of Smalltalk actually did intend to promote code reuse (Pope, Goldberg and Deutsch, 1987), but it seems safe to assume that they did not intend to create search and reuse problems.
Using psychological design rationale. Psychological design rationale can be used as a working design representation throughout the design process. Just as in the case of scenarios, we want to use analyses of existing situations to help set the agenda for design work and to guide reasoning about particular possibilities.
The analysis in Box 3 supported our work on MoleHill by clearly articulating what it is about situations like that depicted in Figure 1 that we should address in our design work. A general hillclimbing heuristic for using a psychological design rationale in this way is to envision alternate scenarios such that downside consequences of particular relations are obviated or mitigated, while upside consequences are enhanced or maintained. Thus the Guru scenario in Box 2 mitigates the downside of the Programming Tools Relation while maintaining its upside. That is, the Guru project post mortem encourages reflection on code design, but does not constrain fluid and opportunistic programming activity prior to the post mortem.
Having placed on our design agenda the goal of encouraging attention and reflection on code design without diminishing flexibility and empowerment, we explored theories and techniques pertaining to reflective activity, and specific designs to promote such activity. Our object was to further elaborate the design of the Guru and its envisioned consequences for users. We studied LISP-CRITIC (Fischer, Lemke, Mastaglio, and Morch, 1990) an experimental critiquing facility to which programmers may submit code fragments to get suggestions about how to make their code easier to read and maintain or more machine efficient. The system does not place any constraints on how programmers create the code submitted for critique, nor on whether or how they use its suggestions. In our analysis, the psychological design rationale for LISP-CRITIC hinges on the possibility that a highly specific critique, delivered at the programmer's request and in the context of a current programming situation, can encourage programmers to reflect more on their work without undermining their preferred work style:
Situated Critique Relation: a situated critique on specific code design problems--at the programmer's request--causes programmers to reflect more on their code without hampering opportunistic code development (but may also cause programmers to reflect on their work too narrowly (may improve a particular code fragment, but fail to teach a more general design lesson).
We had envisioned the Guru as providing a very similar sort of situated critique (though we wanted to address architectural issues, like inheritance --Box 2).
We also investigated the theory and technique of "reflective practicum" (Schoen, 1987) in which a teacher models expert performance for a student, while commenting on the process of producing such a performance and coaching the student on how to move toward achieving such a performance:
Annotated Demonstration Relation: the teacher's self-consciously annotated demonstration of expert performance causes the student to integrate actions and concepts in a paradigmatic model (but may also cause the student to feel coerced (due to the power imbalance between teacher and student)
This kind of scenario and its characteristic relation bears on the goal of helping programmers integrate their understanding of inheritance and the use of "super" in Smalltalk with their strategies for building code. We wished to create the upside consequences of the Annotated Demonstration Relation with the Guru, and believed that we could avoid the downside consequence since the Guru would only be a software tool initiated by the programmer.
Figure 1. Design reasoning for the Smalltalk Guru: The Programming Tools relation highlights the possibility that programmers may not direct enough time and effort to code design. Scenario analysis of the prior system LISP-CRITIC and of the theory and technique of reflective practicum brought to light a variety of issues and design ideas: providing critique within a current problem context, providing narrated demonstrations of target skills to better link concepts and actions, allowing user control of critique initiation and user discretion as to consequent actions. Articulation of the causal relations in these situations guided us in combining their elements in the Smalltalk Guru, a strategic bug critic with demonstration of programming process.
Our experience. Design reasoning is often quite local; designers address sets of requirements and then set aside tentative reasoning and partial results in order to explore other, hopefully converging subproblems (Carroll, Thomas and Malhotra, 1979). This is precisely why it may be useful to create explicit rationale: interim results and their backing can be preserved and deliberatively integrated into the overall design solution. Articulating the causal relations in LISP-CRITIC and reflective practicum situations guided us in combining their elements in the Smalltalk Guru (see Figure 1). This is not to suggest that design reasoning can be reduced to a calculation, rather it is to admit that only when arguments are made explicit can they be evaluated.
In our design work on instructional and programming tools, we have explored the usefulness of articulating the causal relations of a psychological design rationale (Carroll & Rosson, 1991; Carroll & Rosson, 1992; Carroll, Singley & Rosson, 1992; Rosson & Carroll, 1993; Singley & Carroll, 1995; Singley, Carroll & Alpert, 1991). We find that, like scenarios, these representations help us directly by creating auditable design claims about how an envisioned system will affect its users. A clear tradeoff is that while articulating each individual relation of a psychological design rationale creates a more precise representation than a scenario, creating such a rationale is itself a more difficult design task. We have developed a heuristic for generating elaboration questions from which to develop psychological design rationales from scenarios (Carroll & Rosson, 1992) and have found also that having related rationales already in hand is a significant aid; nonetheless developing a psychological design rationale is still a significant task. A practical compromise is to develop a full psychological design rationale only for key scenarios, or for cases where a scenario representation leaves open questions.
It is important to stress that psychological design rationales are heuristic explanations of human behavior and experience. Their utility in system design derives from directly linking features of a designed situation with user activities and experiences. However, this can be an over-simplification: many of the causal relations between features of situations and consequences for people in those situations are not first-order, they derive from interactions and side-effects. Our hope is that when such interactions arise, we might improve our chances of noticing them and of beginning to fathom what's going on if we are keeping explicit account of what we expected would happen. If something is clearly working or failing, we have a basis for making credit/blame attributions, we have a starting point for systematically revisiting our decisions, and for understanding other situations that guided us to those decisions (Carroll, Singley and Rosson, 1992).
This article has sketched a simple but extendable analytic method for managing usability and usefulness, by making use itself a design representation. In this method, user interaction scenarios are a concrete medium for describing existing situations and for projecting envisioned situations. A set of such scenarios is a broadly useful design representation; it is a resource for reasoning about what people will do and experience in a new situation; it is an auditable design claim about what will be delivered to the user. Such a representation can be further elaborated into a psychological design rationale, specifying causal relations among aspects of the designed situation and consequences for people in the situation.
This method attempts to leverage established practices in using scenarios to communicate with users about requirements, to illustrate functional specifications, and to manage usability testing and documentation development. Creating scenarios more systematically and earlier in the design process as planning and reasoning aids is only an incremental elaboration of current practice. If a designer or analyst finds that making more systematic use of scenario representations is truly valuable, he or she may be motivated to consider enhancing these with some causal annotation of why the scenarios run as they do. If this enhancement also seems valuable, he or she may consider developing a psychological design rationale for a project. Hopefully, the gentle gradient from current (somewhat casual) use of scenarios in design practice to the somewhat more systematic use we are developing will enhance the acceptability of the method.
Finally, we have been encouraged that taking a more systematic approach to the representation of use in design can support the cumulation and development of knowledge attained in the course of design. When we can view a scenario as an instance of a type, we are in a position to generalize whatever we learn about that particular scenario to the type. Thus, recognizing that the Smalltalk Guru is a kind of bug critic similar to LISP-CRITIC can serve two purposes: first, it can help us elaborate the design and the design rationale for the Guru, but second, it creates the possibility that things we subsequently learn about the Guru can be mapped back to LISP-CRITIC, or more generally to bug critics, as a specific hypothesis. Similarly, drawing upon general types of human activity--like orienting to goals or evaluating one's actions (Norman, 1986)--in generating scenarios to envision a system in design, can also help us attribute things we learn about given scenarios of use for given systems to a more general analysis of human activity. Inductive generalization is the chief means for creating new knowledge, but it requires a framework for seeing a particular as an instance of a type. Some of our recent work has detailed how the framework of scenarios and psychological design rationale could leverage workaday usability testing as the empirical backbone of a design-based science of human-computer interaction (Carroll, Singley & Rosson, 1992).
We will always need to manage the design of efficient and reliable devices. If we also take seriously seeing them as artifacts of human activity, as elements of designed situations, as participants in causal relations with human experiences, we will improve our chances for deliberately designing computers that are what we want them to be: stimulating and useful tools for people.