Web Comparative Design Tool Evaluation
Maria Cordell Markus Haas Andriy Naborskiyy Oriana Williams
The Design Tools
Dreamweaver
MX
commercially available has extended features for site management
Mozilla
Composer
freeware supports most common Web editing needs
Evaluation Plan
Test
subjects relatively new to design tools Some experience with HTML authoring Tool quality assessment goals
functional completeness learnability pleasantness (lack of frustration)
Devise
consistent test metrics Devise design to encapsulate tasks
Our Process
Six
subjects evaluated using both tools Subject activity captured via Camtasia Passive observation of subjects Questionnaires used to capture quantitative and qualitative data Post-session Camtasia capture studies
Coding Scheme
number number
of times help is accessed of errors number of design omissions (at the end) number of times program help is consulted open existing page as guide time duration for completion of all tasks (overall session time) an "undo" operation idle browsing (hunting) periods number of times code (vs. design) mode used
Questionnaire Results
Question How familiar are with you with Dreamweaver? How familiar are you with the applications in the Macromedia Studio suite? How familiar are you with Mozilla composer? How much experience do you have with creating a web page? How much experience do you have with creating Weblog pages? How comfortable are you in writing HTML code (with no graphical editor or tools)? Familiarity (1 7 scale) 3 3.83 1 4.67 1.17 4.33
Dreamweaver Results
Five
subjects completed tasks in < 18 min One subject did not finish (ran out of time) One subject accessed help User errors: header, image border, text weight, and table height Two subjects started with an existing page Idle browsing: exploring, cell merging, image border, copyright symbol
Composer Results
Three
users did not finish (ran out of time) Three users completed in < 13 min Three subjects accessed help; one was helped User errors: header, page bullet width, list Two subjects started with an existing page Table column resizing, control handles, and horizontal rule problematic
Dreamweaver vs. Composer
Learnability
Total time taken related to familiarity with Macromedia products Composer required trial-and-error for tables Bulleted lists difficult in both Patterns in idle browsing
Functional Completeness
Help system issues in both tools Composer had system bugs or nonstandard UI
Pleasantness (lack of user frustration)
Issues in completeness and nonstandard functionality created frustration One user emitted a grunt of disdain while working with Composer
Conclusions
Users
will transfer gained domain knowledge to a new tool
e.g., methods for table modification, and special character insertion (copyright symbol)
Almost
all users knew how to insert a special character
helped tool learnability, since users instinctively knew where to find a function
Conclusions, part 2
Composer
look & feel led to relatively better performance
predictable text- and formatting-oriented functions; had a learnability advantage
Dreamweaver
had clear advantage in Web page design-specific tasks
setting image border and modifying tables more functionally complete th...