HCI 460

Spring 2016

Craig Miller

Weekly Notes

Assignments

Project 2
Usability Test Plan
Prepare Preliminary Version for Pilot Test (week of April 26)
Final Version Due Sunday May 1 before 11:30 PM

Overview

For this assignment, your group will be preparing a usability test for your assigned application. Your test should approximately correspond to what Rubin calls an assessment test. On a working system, you will be identifying usability problems and determining the amount of user satisfaction and interest for the product. The test plan should be short and simple, thus allowing you to better reflect on its goals, design and limitations. For each participant, you should plan on approximately fifteen minutes of interaction time that, when added to greeting, instructions and debriefing, provides for a half-hour session.

During the week that the test plan is due, all teams will conduct a pilot test using the test plan. Teams conducting their pilot tests on campus will do so on April 28 at a scheduled time. Online teams will conduct a pilot test at their own convenience but will need to provide a commentary outlining issues and changes to the plan. To support the commentary, the online team may post a video of the pilot test. The commentary may be completed through a phone conversation with the instructor.

Planning the usability test

In chapter 5, Rubin provides a model format (outline on p. 67) for a usability test plan. You should also follow this format in creating your plan. The following provides additional requirements, guidelines and suggestions for completing each stage.

Purpose

This statement may be one or two sentences that addresses the need to determine the application's usability problems and interest to potential users. You are welcome to invent some story as to why your group would want to do this (e.g. your company might want to buy and use this product, or the application's developers have asked you to evaluate it).

Problem statement and test objectives (called research questions in Rubin)

In this section, you provide the specifics of what your group will be looking for. The heuristic evaluation and the cognitive walkthrough can help you provide these specifics. For your plan, list five to eight test objectives in the form of questions, one to two sentences each. At least one of these objectives needs to address the user satisfaction of the product.

Identifying good test objectives is critical for planning and conducting usability tests. See Figure 5.2 for examples. I strongly encourage you to discuss your proposed research questions with me before you proceed with the rest of your planning.

User profile

The user profile should be consistent with the context of use as presented by the representative from your assigned organization. In most cases, you'll be able to make reasonable inferences for what your users know and can do. If required, you can contact the organization representative for any clarifications.

Test design

Keep the design simple. When describing your procedure (which includes greeting, instructions, debriefing, etc.), refer to scripts and checklists, but append these materials to the end of the test plan. Start with a short overview of your design (a few sentences) before writing out the details (a few paragraphs). Make sure you specify the testing roles of your group members.

In addition to one pilot run, your group should plan to run tests with at least four participants. Ideally, some of your recruits will come from outside of class.

Task list

The set of tasks should last approximately 15 minutes. It may be one long task or a set of small tasks. If it is several tasks, state and motivate their order.

Test environment

A short paragraph should be adequate here. Consider what space will be conveniently available to your group. In all likelihood, you will describe something similar to Rubin's simple single-room setup, possibly without the video camera. For the live section, we are planning to conduct the pilot test in the class room. All other tests can be conducted in a location convenient to your group.

Test monitor role

Here your group determines and explains how much the test monitor interacts with and helps the participant.

Evaluation measures

Decide what data your group will record in order to address the test's objectives. The data may be qualitative or quantitative. Creating and submitting a blank log sheet would be a good addition to your test plan, which you could place in the appendix and summarize in this section. Evaluating the user's satisfaction with the product will probably involve a post-test questionnaire.

Report contents

This can be a summary paragraph anticipating what conclusions your final report will address. It includes explaining how the evaluation measures will address the test objectives.

Supplemental materials

These include consent forms, scripts, questionnaires, checklists and blank log sheets. They need to be prepared and turned as an appendix to the test plan.

Submission

Submit your assignment using the D2L web site. PDF is my preferred format, but I will accept any commonly available format (e.g. Word, HTML). The report should include the following:

  • The usability test plan including all sections and supplemental materials.
  • The names and email addresses of the group members and their principal roles in the assignment.
  • A statement or key indicating how the plan was revised based on the pilot test.

Grading

A D2L rubric will be used to assess the report.