Methodology

 

This project deals with the popular strategy game Warcraft, and it consists of both a theoretical analysis and an experimental study. The study was conducted in two parts. In part I of the project, simple user tasks of the game were identified and GOMS and TAG analyses were conducted to reveal the usability issues underlying the game. In part II, user scenarios were developed and an experiment was conducted using both novice and expert users. Detailed claims analysis was done from the user experiments.

Part I: GOMS and TAG analysis

GOMS analysis was conducted to develop keystroke-level estimates of the time needed to perform various elementary tasks in the game. The main input device in the game was the mouse, but there are also keyboard shortcuts intended for advanced users. The analysis compared various primitive tasks in terms of the number of keystrokes needed to perform the task.

Additionally, Task-Action Grammar (TAG) analysis was performed on selected tasks in the game. Consistency across TAG of various user tasks was analyzed to reveal the underlying learnability issues.

Part II: User experiment with novice and expert users

This is a detailed experiment involving novice and expert users. To conduct this experiment, we developed custom scenarios. Each novice user was asked to play novice scenarios in both Warcraft and Warcraft II, and each expert user was asked to play expert scenarios in both Warcraft and Warcraft II. Part II of the experiment was further divided into three sections, the pilot study, questionnaire and interview, and the actual experiment.

a. Pilot study:

To test the custom scenarios we developed, a pilot study was conducted with one novice user and one advanced user. The pilot study was conducted during the last week of October 1998. Based on the results of the test run, we modified certain shortcomings of our scenarios.

b. Questionnaire and interview:

Questionnaire and interview techniques were used to get each user's background, feedback after playing the game, and the level of motivation generated by the game. A pre-test questionnaire was used to assess a user's background as well as previous experience with computer games, and a post-test questionnaire was used to collect the feedback from the users and assess the level of motivation. The user responses were quantified using a pre-structured scale. Additionally, to analyze the motivation provided by the game to novice users, a long-term follow-up questionnaire was used to evaluate the response of novice users to the game after a period of three to four weeks after the experiment.

In addition to the questionnaires, detailed interviews were conducted to assess user motivation qualitatively. The interviews also addressed issues that are not covered by the questionnaire.

c. User Experiments:

The user experiments were conducted during the first week of November 1998 with four novice users and three expert users. Custom scenarios were developed for both novice and expert users for both versions of Warcraft.

Data analysis:

The generated by the questionnaire were analyzed using tabular and percentile analysis. The qualitative data generated from the interview were used for the analysis of the results of the study.