Results of Novice user scenarios

  

 

Profile of Novice Users


The four novice subjects that participated in our study had a wide range of backgrounds. None of them had played either Warcraft or Warcraft II, but they had varying game-playing experience. Of the four subjects, only one played a lot of computer games, but mostly real-time simulation games (e.g. flight simulators). The other three had limited experience in playing computer games. None of the subjects had extensive experience with other real-time strategy games, but two had limited experience with these types of games.

Pre-study Survey Results

Before beginning the scenario, each subject was asked to fill out a survey about their game-playing experience. The average results of the survey were (1-Strongly Disagree, 2-Disagree, 3-No Opinion, 4-Agree, 5-Strongly Agree):
 
 
 
 
Question
Average Response
I am an experienced computer game player.
2.25
I play a lot of computer strategy games.
1.5
I use a computer every day for my profession.
4.75

 

Post-study Survey Results

After the scenario, we asked each subject to fill out another survey about their impressions about the games. The average results of the survey were (1 - Worst, 5 - Best):
 
 
 
Category
Warcraft I Average Rating
Warcraft II Average Rating
Ease of use.
3.33
3
Easy to learn.
3.67
3
Fun to play.
4
4

Post-study Interview

After each novice user completed their scenarios, we held a quick exit interview. The interview had two main purposes. First, we wanted to judge the amount of engagement the user felt with the scenario. Secondly, we tried to determine the user's level of motivation to continue playing. It was fairly easy to determine the users level of engagement. (See User Engagement). However, trying to determine what motivated the users was difficult. During the interview it seemed as if all of the four subjects were interested in playing Warcraft again. It also seemed like these people were the ones who were the most involved with the scenarios.

It makes sense that there would be a correlation between the amount of engagement that a user felt and their likelihood to continue playing. This engagement that most of the users felt is also the feeling of escape that many people are looking for in computer games. Unfortunately, the short scope of our study prevented us from investigating this further.

Follow-up Questionnaire on Motivation

Several weeks after the end of the study we sent a new questionnaire to the subjects to see if they had a continued interest to play the Warcraft games. The idea of this was to see if the motivation that users felt was ephemeral (short term) or sustained. The results of the questionnaire were (1-Strongly Disagree, 2-Disagree, 3-No Opinion, 4-Agree, 5-Strongly Agree):
 
 
 
Question Average response
Played Warcraft since study  3
Interested in playing since study  3
No time to play since study  4.5

None of our subjects had played Warcraft since the study. However, two of the four expressed an interest in playing again. All of them cited a lack of time as the main reason for not playing. This shows that most of the users had a continuing motivation to play the Warcraft games. However, none of the subjects had a very high level of motivation, since none of them were planning to buy the game.

Interface Problems Found

The main problems that we found the novice users had were:

Useful Features of Interface

The best features of the interface for the novice user were:

Unexplored Areas

Both of the Warcraft games use the same metaphor for unexplored areas. In the games these areas initially appear as black areas (See Screen Shot ). Then when the player moves a unit into the area, the area clears up, and the terrain in the area is visible. This is supposed to create a metaphor of explorers creating a map as they explore an area. However, this metaphor leverages on prior knowledge that isn't present in some people. Most strategy games use this same metaphor, but it can be confusing the first time a user encounters it. In out study, two if the users had encountered this metaphor in prior games and had no trouble understanding it. However our other users, were confused by this metaphor.

Training Procedures

All of the novice users had trouble discovering how to "train" peasants.  Both versions of Warcraft call the creation of new units "training". To create a new unit the user must select the appropriate building that creates these units. With this building selected, the user can "train" a new unit and have a new unit appear. Most of the confusion arose with the term "training". When faced with the task of "training" a new unit, all of the novice users selected a unit of that type and looked for an option to train that unit. Two of the users became so frustrated with this task that they finally moved on to the next task. However, all of the users eventually found the correct method for creating new units. Then, once the novice users discovered how to train a new unit of one type they quickly realized that the same method could be used for all of the units.

Lack of Feedback

The Warcraft game interfaces are usually very good about providing feedback about the progress toward completion of user assigned tasks. For example, when the user selects a building that is under construction, the system displays a progress bar showing the status of the construction (See Screen Shot). However, there is one notable task where the system does not provide the user with feedback. When a peasant harvests lumber no feedback is provided. Harvesting lumber is a very common task that can take a significant length of time. Many times a novice user has to wait for this task to complete, but there is no indication of how this task is progressing. This lack of feedback is inconsistent with the rest of Warcraft, and can be frustrating to a user.

Lack of Help

All of our test subjects took a very active approach to learning how to use the interface. None of them consulted the manual before they started using the interface. They all jumped right in and started trying out different things. Most of the features of the interface were discovered by selecting the various units and looking at the selections that became available. Both of the Warcraft games lend themselves well to this type of exploration. However, when the user runs into problems, there is very little help that they can get. In both games the on-line help consists of a list of keyboard commands that are open to the user. Warcraft II provides a tip that appears at the beginning of each scenario. Some of these tips are useful, but most often they apply to shortcuts that expert users can use to save time. Warcraft II does provide the ability to display a tip during game play, but there is no organization to the tips and no way to search them. Two of the novice users attempted to use the on-line help, and neither found it to be of any use. One user became so frustrated that they attempted to use the manual, but that failed to be of any more use. Overall, a novice user of either of the versions of Warcraft is on their own.

Prompting on Unavailable Actions

In Warcraft, when the user attempts a task that is not allowed, the system responds with a message (See Screen Shot). This message tells the user why the attempted action was not allowed, and also gives a suggestion on how to correct the problem. For example, if the user attempts to train a unit when food supplies are low, the system will respond with "Not enough food.... Build more farms." By providing this information the Warcraft uses distributed cognition to provide the user with just the information necessary to correct the problem. All of the novice users liked this set up, and two specifically commented on it.

Status Panel

Users also appreciated the way the status panel would change depending upon the type of unit selected. In Warcraft, when a unit is selected, the options corresponding to that unit appear in the status bar (See Screen Shot). This is another example of distributed cognition that both Warcraft interfaces use to make the system more usable. Here, the system provides the user with just the information necessary, and this limits the number of possible choices for the user. The user doesn't have to search through a large set of possible actions, the system limited only displayed the useful choices. The novice users found this especially useful when exploring around. Limiting the number of choices available made it likely that the novice user would eventually try a useful task. In our scenarios all of the novice users discovered how to "train" a peasant in this way. The status bar was an excellent tool to help an active user explore the interface, and learn it in their own way.

Overall Map

The system also provides a small map that shows a thumbnail view of the entire game area (See Screen Shot). Once the novice users discovered this they found it an extremely useful function. With this it is easier for the user to get an overall view of the game area. The user can also click on the smaller map to move the larger view to that section. The user can also command units to move to a distant location using the overall map. The method for doing this is consistent with moving units on the smaller screen, and is fairly simple for the user to determine. Half of our novice users discovered this with no prompting.

User Engagement

One of the most interesting things that we noticed about the novice users was the extent to which they became engaged with the scenario. Some of our novice users stopped cutting down trees and mining gold after they had the minimum amount required. When we asked them about this in the exit interview, they talked about not harming the environment any more than necessary. Also during this interview some users expressed concern about their units being killed. Both of these things point to the fact that the user is drawn into the Warcraft environment. Their are several things that contribute to this. First is the direct manipulation flavor of the interface. The changes made by the user take immediate affect, and feedback from these commands is instantaneous. Secondly, the interface very quickly becomes transparent. The interface quickly becomes natural even to a novice user. We noticed that by the second scenario all of the novice users had no trouble in moving their units.

Screen Shot Example