Traditional vs. Online Testing

• Some traditional forms of test submission:
  • Letter (long delays, used in distance learning)
  • On-site: Paper (long turn around time in large classes, most commonly used)
  • On-site: Oral (more interactive but very time consuming)

Traditional vs. Online Testing (cont.)

• E-mail
  • Much faster than letter, but still long delays in grading
• Computer-Based Training/Testing Systems (CBTs)
  • Examples: Authorware, Toolbook, GRE
  • Automatic Grading, Instant Feedback
  • Platform dependent, hard to integrate record keeping

Existing WWW-Based Testing Systems (WBTs)

• Advantages over CBTs
  • Platform independent
  • Increasingly popular in universities
  • Affordable
  • WWW server centralizes requests (server-side grading and record keeping)

Existing WWW-Based Testing Systems (cont.)

• Disadvantages
  • Authoring languages are HTML extensions
    • Focus on presentation as opposed to structure
  • Minimum database and record keeping support
  • Lack of essential security features
  • Tests are not adaptive (i.e., test contents are fixed and do not depend on student performance)

The QUIZIT System: Overview

• Instructor Module
  • Content-based test representation
    • The QUIZIT Markup Language (QML)
    • HTML translation
  • Central database, extensive record keeping
    • Database Monitor WWW-based Interface
    • MSQL (Hughes Corp.)

The QUIZIT System: Overview (cont.)

• Student Module
  • Security features:
    • Authentication
  • Instant feedback on adaptive tests
    • The Grader Engine
  • Reviewing old quizzes and records
    • Review Engine

The QUIZIT Instructor Module: QML

• Content-based representation
  • Use ofSGML (ISO/IEC 8879) to define QML
  • Avoids presentational elements

The QUIZIT Instructor Module: QML translation

• After Instructor prepares the QML file

The QUIZIT Instructor Module: Database Gateways

• Log student accesses to the system
• Keep track of individual and class performance on quizzes
• Can change system state and db. at any time if user is authorized to do so (instructor, GTA)

MSQL Database Engine

• Stores and Retrieves students' records and accesses to the system
  • IDs and passwords
  • Scores and student's answers on every quiz (for every unit in the course)
  • Access logs (begin, end time)
  • Last quiz taken
  • Next quiz to be taken

The QUIZIT Student Module: Authentication

• Provides online password registration and distribution (by Tommy Johnson)
  • Passwords are encrypted in the server side (is client side encryption feasible?)
  • Users can change their own password at any time
• Authenticates every user once and then uses session cookie throughout the quiz (use of hidden fields)

The QUIZIT Student Module: The Grader Engine

• After selecting a course, authenticating, selecting a unit and taking the quiz...

The QUIZIT Student Module:
Reviewing Records and Old-Quizzes

• At any given point in time students can:
  • Review old quizzes (student may not retake quizzes for credit)
  • Come back to the system and re-start from where they left
  • Review their records
    • submission times
    • scores (cumulative, per quiz)
    • the next quiz they will have to take

QUIZIT in action!

• Graduate-level course: CS5604 (about 20 students)
  • Personalized System of Instruction (PSI)
    • Course divided into 12 units
    • Each unit has one quiz with two levels. Each level has two remedial quizzes

QUIZIT in action! (cont.)

• Undergraduate Level Course: CS1604 (about 200 students)
  • Each quiz has just one level
  • Security concerns are much greater
    • Password distribution needs to be automated
    • How can we assure that one student is not taking a quiz for another?

Evaluation and Conclusion (HCI aspects)

• Authoring quizzes with QUIZIT takes the same amount of time as traditional quizzes
• Huge savings in grading time, though...
• Security and authentication are critical in large enrollment classes
• Extensive record keeping and WWW monitor interfaces are very important
• Change in the role of the GTA

Evaluation and Conclusion: Technical Aspects

• Instructors need training in SGML
• CGI/MSQL is a very nice combination
• Record keeping can be further extended
• Further security mechanisms need to be developed (client side encryption, camera monitoring, etc..)

Future Developments

• Improvements in student feedback
  • Better error report after each test
• Graphical User Authoring Interface
  • but keep QML representation
• Improve question modeling
  • Add more pattern matching options for fill-in questions
• Develop more scoring methods