Performance Measurements: Email | Performance Measurements: Web Browsing

Voice vs. keyboard/mouse control
Expert Performance measurements

Performance Measurements: Email

Methodology

• task from scenario 1 (but Microsoft Outlook instead of Eudora):
  short email reply to the following message:
  "Hi Tom, did you read the book I told you about last week?"
• minimal cognitive effort to read and understand the email (3 seconds)
• the reply-text:
  
  "Hi Frank,
  Today I finished the book. I really enjoyed reading it.
  If you find something like that again let me know.
  
  Thanks, Tom"

• after a period of training, the expert performed the task 10 times using the voice recognition software and 10 times using keyboard & mouse
• all errors were counted
• expert: familiar with Microsoft Outlook (mouse control as well as menu structure, 1 year experience),
  familiar with DragonDictate (3 weeks daily experience), typing performance: 40 words per minute

Voice strategy

• start outlook ([Bring Up Mail])
• move to new message ([Ctrl Key][uniform])
• read displayed message
• compose reply ([Compose] [Romeo] [Dictate Mode])
• dictate reply text:
  "Hi Frank[Comma][New Paragraph]
  Today I finished the book[Period] I really enjoyed reading it[Period][New Line]
  If you find something like that again let me know[Period][New Line]
  [New Line]
  Thanks[Comma] Tom"

• send message ([Command Mode] [File] [Enter])
• create new folder ([Edit] [Mike] [Oscar] [Alt key] [November] [Type Word] [Friends] [Enter])
• move old message to new folder ([Foxtrot] [Enter])
• return to previous application ([Bring Up Microsoft Word])

Keyboard & mouse strategy

• start outlook (mouse-click on outlook icon in taskbar)
• move to new message (mouse-click on highlighted message)
• read displayed message
• compose reply (mouse-click on "Reply to Author"-icon in toolbar)
• type text ("Hi Frank,...")
• send message (mouse-click on "Send"-icon in toolbar)
• create new folder (right mouse-click on "Outlook-Express", select "New Folder...", enter folder name, press Enter key)
• move old message to new folder (drag message to newly created folder)
• return to previous application (mouse-click on Winword icon in taskbar)

Measurements

Types of errors:

• Recognition errors: DD misinterpreted a word or command
• Navigation errors: Expert issued wrong command (eg. saying [File] before entering [Command Mode])
• Typographic: typing wrong letter(s)

Comments

• voice: best time was 86 seconds (user defined macros could improve performance)
• keyboard & mouse: best time was 48 seconds
• almost no navigation errors while using keyboard & mouse
• more navigation errors using voice control (use of menu structure requires more cognitive effort than use of the toolbar)
• voice recognition errors occur even after thorough voice training (error occurred while breathing or pronouncing words in a slightly different way)
• keyboard typing errors do not have a significant influence on performance
• keyboard/mouse method approximately 100% faster than voice recognition

Performance Measurements: Web Browsing

Methodology

To compare the performance of voice control and speech control, we measured the time each method required to perform the following task:

Open a web browser and navigate to Yahoo’s search page. Search for the Adobe home page, navigate to the page, complete a registration form, and initiate a download of the Acrobat reader program.

For each method, we performed the task ten times and recorded time required along with relevant errors. A recognition error occurred when DragonDictate incorrectly recognized a word and typed it or incorrectly recognized a command and executed it. A navigation error occurred whenever the user incorrectly directed the mouse pointer. A typographical error occurred whenever the user misspelled a word.

Voice strategy

• Give command to [Bring Up Internet Explorer]
• Give command to [Go To Yahoo]
• Enter the search query: "Acrobat reader download" while in [Dictate Mode]
• Navigate to Adobe home page link via [Mouse Grid] control
• Navigate to registration page via [Mouse Grid] control
• Complete the following fields of the registration form: Name, Address, Company, City, Zip, State, Email address
• Submit the form
• Click the download link

Keyboard & mouse strategy

• Click on Internet Explorer icon on desktop
• Enter the search page URL: "www.yahoo.com"
• Enter the search query: "Acrobat reader download"
• Navigate to Adobe home page link
• Navigate to registration page
• Complete the following fields of the registration form: Name, Address, Company, City, Zip, State, Email address
• Submit the form
• Click the download link

Measurements

Comments

We discovered two main bottlenecks with the voice control: mouse movement and form completion. Mouse control is done via a numbered tic-tac-toe grid on the screen. Calling a number moves the pointer to the center of the specified sector which is then subdivided into nine more sectors. This process continues until the pointer reaches the desired location. Every mouse movement must be controlled in this cumbersome manner. Form completion was hindered by the necessity of swapping between [Command Mode] and [Dictate Mode]. This was necessary in order to move between fields with the [Tab] command.

Another significant factor in the speed of the voice control was recognition. A quick glance at the data shows that every trial with a time exceeding 200 seconds had at least 3 recognition errors. In contrast to this, typographical errors had little overall effect on the times measured with mouse and keyboard control. This seems to indicate that voice recognition errors are more time-consuming than typographical errors.

In general, the Internet task was performed over twice as fast with the keyboard and mouse control. This method also generated more consistent results than the voice control. The inconsistent times of the voice control are to be expected since error recovery is more time-consuming.

Theories Proposals