3DUIevalI(1) - 3D User Interface Evaluation I 3D Lecture...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: 3D User Interface Evaluation I 3D Lecture #15: Evaluating 3DUIs – Part I Spring 2009 Joseph J. LaViola Jr. Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. User Evaluation in 3DUIs Was missing component for many years Was novelty novelty limitless possibilities limitless exploration of design space exploration Field has matured Field Need to compare Need devices devices interaction techniques interaction applications applications etc… etc… Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 1 Purposes of Evaluation Evaluation – analysis, assessment, and Evaluation testing of an artifact Problem identification and redesign Problem General usability understanding General Performance models Performance Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Some Terminology Usability – everything about an artifact Usability and what affect a person’s use of an artifact Evaluator – person who designs, Evaluator administers, implements, or analyzes an evaluation Subject – person who takes part in the Subject evaluation Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 2 Evaluation Tools User task analysis User generates list of detailed task descriptions, sequences, user generates work, and information flow Scenarios Scenarios built from task analysis built important for experiment design important Taxonomy Taxonomy science of classification science break techniques into components break used in evaluation process used Prototyping Prototyping need to have something to test need paper-based sketches paperWizard of Oz approach Wizard Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Evaluation Methods Cognitive walkthrough Cognitive Heuristic evaluation Heuristic Formative evaluation Formative observational user studies observational questionnaires, interviews questionnaires, Summative evaluation Summative task-based usability evaluation task formal experimentation formal Sequential evaluation Testbed evaluation Questionnaires Questionnaires Interviews and Demos Interviews Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 3 Evaluation Classification Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Evaluation Metrics – System Performance System performance metrics System Avg. frame rate (fps) Avg. Avg. latency / lag (msec) Avg. (msec) Variability in frame rate / lag Variability Network delay Network Distortion Distortion Only important for its effects on user performance / Only preference frame rate affects presence frame net delay affects collaboration net Necessary, but not sufficient Necessary, Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 4 Evaluation Metrics – Task Performance Speed / efficiency Speed Accuracy Accuracy Domain-specific metrics Domain education: learning education: training: spatial awareness training: design: expressiveness design: Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Subjects will make a Subjects decision Must explicitly look Must at particular points on the curve Manage tradeoff Manage Accuracy Speed-Accuracy Tradeoff Speed Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 5 Evaluation Metrics – User Preference Ease of use / learning Ease Presence Presence User comfort User Usually subjective (measured in Usually questionnaires, interviews) Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. User Preference in the Interface UI goals UI ease of use ease ease of learning ease affordances affordances unobtrusiveness unobtrusiveness etc. etc. Spring 2009 Achieving these goals Achieving leads to usability usability Crucial for effective Crucial applications CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 6 User Comfort Simulator sickness Simulator Aftereffects of VE exposure Aftereffects Arm/hand strain Arm/hand Eye strain Eye Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Measuring User Comfort Rating scales Rating Questionnaires Questionnaires Kennedy - SSQ Kennedy Objective measures Objective Stanney - measuring aftereffects Stanney Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 7 Characteristics of 3DUI Evaluation Physical environment Physical Evaluator issues Evaluator User issues User Evaluation type issues Evaluation Misc. issues Misc. Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Physical Environment Issues Utilizes nontraditional input and output Utilizes devices Many displace do not allow multiple Many simultaneous viewers Think-aloud and voice recognition Think Mobility and video recording Mobility Collaborative UIs and network behavior Collaborative Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 8 Evaluator Issues May require more than one May Breaking presence Breaking No evaluator intervention means robust No software instructions must be detailed instructions Challenges with multimodal interfaces Challenges Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. User Issues Selection of subject pool Selection 3DUIs may not be well understood 3DUIs Novice vs. expert users Novice Number of subjected needed may be Number larger than normal (novelty) Users must adapt to wide variety of Users situations Effects of cybersickness Effects Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 9 Evaluation Type Issues Heuristic evaluation difficult due to lack of Heuristic guidelines Not many performance models for 3DUIs Not Automated tools are important Automated not many of them for 3DUIs not Multi-attribute Usability Evaluation Tool for Virtual MultiEnvironments (MAUVE) – Stanney et al. 2000 Statistical validity and 3DUI hardware Statistical many factors to consider many Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Miscellaneous Issues Focus at a lower level Focus difficult to evaluate on application level difficult no set 3DUI standards no Generalization of results Generalization Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 10 Next Class 3DUI Evaluation cont’d 3DUI Readings Readings 3DUI Book – Chapter 11, 349-367 3DUI 349- Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 11 ...
View Full Document

This note was uploaded on 06/13/2011 for the course CAP 6938 taught by Professor Staff during the Spring '08 term at University of Central Florida.

Ask a homework question - tutors are online