Scenario-based Claims Analysis

Note: names have been changed to protect individual privacy

Scenario One

Getting Started with the First Presenation

Sam has just begun working with the Presentations for the first time. The instructor has given the class a brief overview, including a short demonstration, and has directed students to more information should it be needed. Sam has gone through the homepage and opened the Presentation for Section 1.1. He sees a new, smaller window in front of his browser, and the status bar on his desktop (Sam is on a PC) indicates that the Authorware plug-in has loaded "p11b.aam". The first screen is entitled "Index to Main Topics of Section 1.1" and there is a numerical listing of topics. When Sam moves his cursor over the text for the topic called "Introduction", the cursor changes from a pointer to a pointing hand. Sam clicks once and he is taken to the first page that briefly introduces what will be covered. When he is done reading the first page, Sam clicks the button labeled "Next" located at the far right of the navigational toolbar at the bottom of the window. Sam continues in this manner, clicking Next to advance and "Back" (which is located at the far left on the toolbar) when he wants to see a page again. As he proceeds, Sam tries the demos in the Presentation, which occasionally give him trouble when he tries to repeat them (this point is elaborated in a later scenario).

This scenario was chosen to be first not only because it covers an introductory situation, but also because it points out many key artifacts that are consistently maintained in all presentations. This scenario introduces major artifacts of the interface and lends itself to an analysis based on Shneiderman's 8 propeties of Direct Manipulation.

1. Continuous representation of the object of interest : The basic window is always visible, with the navigational toolbar at the bottom of every page (except the Introduction).

2. Physical actions or labeled button presses used : Virtually every aspect of the interface satisfies this property. Labeled buttons are the fundamental artifacts of the interface. Each button also has a relevant graphic. Many button actions have corresponding keyboard commands.

3. Rapid, incremental, reversible operations whose impact on the object of interest is immediately visible: Presentations are designed to be dominantly procedural and incremental. Multiple button clicks allow rapid and reversible changes in direction. With the exception of demos we found to be slightly flawed, demos are easily repeated by clicking on them once.

4. Novices can learn basic functionality quickly, usually through demonstration by a more experienced user: This is reflected in the scenario. Sam's teacher gave a brief demonstration to get him started. This behavior was well supported by the original 12 test subjects. After a few words about the presentations and a very brief demonstration, the subjects reported few problems in using Presentations.

5. Experts can work extremely rapidly to carry out a wide range of tasks, even defining new functions and features This component is not directly applicable to the scenario. However, the capability exists for a student to return quickly and easily to an arbitrary location in an arbitrary Presentation. Additional functions and features cannot be created, but students were observed streamlining their methods to make use of a particular set of the buttons.

6. Knowledgeable intermittent users can retain operational concepts: The actor in this scenario who exhibits this behavior is the instructor. In many cases, instructors only accessed Presentations to edit them during design or familiarize themselves with the material. Very often several instructors were intermittently involved in editing, leaving the task for more important design work, returning to it days or weeks later.

7. Error messages are rarely needed: Error messages are never needed.

8. Users can see immediately if their actions are furthering their goals, and if not, they can simply change the direction of their activity : Since the goal of the Presentations is not to do work, but to learn what is presented, only low level tasks such as pressing the "Next" button or starting a demo can be analyzed in this way. The only factor which might inhibit this property is the speed at which the server updates the local machine whenever a key press causes a new code segment to be downloaded.
Scenario Two

The Demo with Animation (moving graphics)

Chris is using one of the online Presentations, when he comes across a particular type of demo. He recognizes the familiar "pointing hand" with the word "Demo" beneath and the dark grey action box that are used on all interactive demos like this one (there are other types). When he positions the mouse over the grey box, the standard "arrow" cursor changes to a pointing hand as well. Chris clicks on the demo and entries from a matrix are animated across the screen, as if some operation is about to take place on the matrix. At this point, Chris has decided that he would like to see this demo again. He isn't sure he quite followed the whole thing.

Chris recalls, from previous experience, that the demos can be repeated by simply clicking on the grey box again. This time, however, something unusual happens. Only parts of the screen are erased, leaving all of the graphics that had been animated still on the screen. (In other demos, the whole screen had been erased.) The graphics that had previously been animated do not return to their start positions and instead remain static.

This scenario captures a typical error situation for almost all users, but it also serves to point out some strengths in the interface.

Strengths

The familiar "pointing hand" graphic with the word "Demo" beneath, and the consistent use of the grey action box are strong cues of what the expected action is.
The change in the cursor (from pointer to hand) serves as positive reinforcement of the action.
The animation in the first run of the demo is a good interface metaphor for an instructor drawing on the chalkboard.

Weaknesses

The instructor typically supports his/her use of the chalkboard with a verbal explanation of what is taking place.
The partial resetting of the demo is frustrating and further confuses students who were unsure of what had just taken place.
There appears to be no satisfactory way to reset the demo.

In the above scenario, students were able to work around this error by selecting the "Back" button, followed by selecting the "Next" button, which would bring the student back to the original spot. Since the error did not always happen, the programmers had to look into thier source, where they discovered an inconsistency in how they constructed the screen layout.

Scenario Three

Use of the "More" Button in Demos

Laura has used the "More" button to advance several steps into a demo when she realizes that she missed a crucial point that is no longer visible. Since the information was just on the last screen, Laura reasons that she should just click on the "Return" button to step back. When Laura does this, however, she is taken back to the section just before the demo.

This is a typical scenario for demonstrations that span several pages. We found that clicking on the "Back" button instead of the "Return" button creates the same scenario.

Weaknesses

The demo is only procedural in one direction. It can be "stepped into" a single frame at a time, but any attempt to retrace the previous step causes the user to leave the entire point of reference.
The Presentations are intended to be entirely procedural (linear order), but the addition of the "More" button introduces a dimension of depth and leads to disorientation.
Some students might miss important parts of a demonstration or section.

One solution being discussed is to remove the "More" button entirely from demos that span several pages, leaving only "Next" and "Back". This would decrease the number of possible choices that seem to lead into confusion and would allow students to step incrementally in both the forward and backward directions. Interactive demos that could not fit on one page would require a new method of presentation. This, however, could create a problem for the designers.

A major advantage of using the "More" button is that it allows programmers to add to a page without having to redraw the entire page or duplicate images within the layout framework (much less code). Since the Presentations are delivered over the Internet, it is currently vital to keep file size down (which in turn keeps transmission speed up). For Presentations with many demos, file size could potentially double and the end user would experience slower overall transmission rates.

Another possibility is to override the action of the "Next" button whenver more information is going to be added to a page. The "Next" button can resume its normal function when the display is completed; advancing to the next page.

Scenario Four

Logging Into a Quiz

Sue wants to take a practice quiz for section 1.1 on her PC at home. She goes to the syllabus and clicks on the link which takes her to that section. She clicks on the button labeled "Quiz" and an alert dialogue box tells her that the Quiz is loading. She clicks on the OK button and a Navigator window, entitled "Linear Algebra Hyberbook" appears on her screen with 3 frames and no buttons. Two frames display the standard error for files not found and one shows a link labeled "Show Index". The Quiz does not seem to be anywhere on her screen. Sue clicks the link, which changes to "Show Contents", but nothing else seems to happen. Sue closes the bogus window and by chance, notices that the quiz appears on her status bar. She clicks on the status bar and the quiz interface maximizes on her screen. The instructions say, "Enter you PID and password below." But only one text-entry box, labeled "PID" is on the screen. Immediately, Sue questions where she is supposed to enter her password. Unsure of what is taking place, Sue types in her PID and hits "Enter". When she does this, another text box appears with space to enter her password. She enters her password and is able to continue.

Weaknesses

The immediate implication is that the wrong action has taken place and that the quiz has not downloaded.
The missing Hyperbook files are an obvious confusion to the user. Further, since the "Go To Reference" button is available within each quiz, it seems unnecessary to bring the Hyberbook up by default.
The quiz should appear on top of all other windows so that the user knows the intended action has indeed taken place.
Both text-entry boxes should be immediately visible in order to remain consistent with the instructions.

Scenario Five

Using the "Skip Questions" button in Quizzes

Jack is taking a practice quiz when he comes across a question he isn't quite sure how to answer so he decides to skip it and come back to it later. He selects the button marked "Skip Question" at the bottom of the quiz interface and he is prompted to confirm his decision by selecting a button called "Confirm Quit". When the confirmation button is highlighted, all other choices are temporarily grayed out. Jack confirms his decision and advances to the next question on a new "page". The new page does not reveal any buttons indicating that it is possible to go back to the previous question; only the standard three ("Quit", "Go to Reference", and "Skip Question"). Jack continues taking the quiz when he suddenly reaches the statistical summary of his performance, which indicates that he has skipped questions. The only button on the page is labelled "Continue" and when Jack selects it the quiz quits, leaving Jack frustrated as to why he could not go back over his answers.

Many students indicated a desire to be able to check their answers in some way. As this scenario points out, students are permitted to skip questions without any warning of consequence.

Strengths

Providing the ability to advance past tough questions is important to maintaining a "self-paced" policy. ¹
The confirmation of an action is a good way to avoid a common error caused by inadvertant mouse clicks.
The graying-out of other choices during a critical decision focuses the student on the action.

Weaknesses

There is no warning indicating that skipping a question will cause a penalty.
Questions are not numbered and the student suddenly finds himself at the end. Numbering the questions would provide important reference points.
There are no opportunities to review answers before they are "submitted".

In this scenario, there are two things that could be added to the display and decision making process that might enhance the entire delivery of all quizzes. First, a reminder would be helpful, at the beginning of each quiz, to indicate that unanswered questions are penalized. Second, a method is necessary that allows students to go to any single question at any given time. This would allow students to review their answers for accuracy and to return to unanswered questions when the reach the end of the quiz but have time remaining to go back.

¹ One of the features that is particularly important to the designers of this system is that students must be able to work at a pace that suits their needs. Flexibility in this way is a key issue.