Mining the Web for Design Guidelines

Marti Hearst, Melody Ivory, Rashmi Sinha UC Berkeley

The Usability Gap

196M new Web sites in the next 5 years [Nielsen99] ~20,000 user interface professionals
[Nielson99]

The Usability Gap

196M new Web sites in the next 5 years [Nielsen99] A shortage of user interface professionals
[Nielson99]

Most sites have inadequate usability [Forrester, Spool, Hurst]

(users cant find what they want 3966% of the time)

One Solution: Design Guidelines

Example Design Guidelines

Break the text up to facilitate scanning Dont clutter the page Reduce the number of links that must be followed to find information Be consistent

Problems with Design Guidelines

Guidelines are helpful, but

There are MANY usability guidelines

Survey of 21 web guidelines found little overlap [Ratner et al. 96] Why?

One idea: because they are not empirically validated

Sometimes imprecise Sometimes conflict

Question: How can we identify characteristics of good websites on a large scale?

Question: How can we turn these characteristics into empirically validated guidelines?

Conduct Usability Studies: Hard to do on a large scale Find a corpus of websites already identified as good!

Use the WebbyAwards database

Talk Outline
WebbyAwards 2000 Study I: Qualities of highly rated websites Study II: Empirically validated design guidelines Putting this into use

Criteria for submission to the WebbyAwards

Anyone who has a current, live website Should be accessible to the general public Should be predominantly in English No limit to the number of entries that each person can make

Site Category

Sites must fit into at least one of 27 categories. For example:

Arts Activism Fashion Health News Radio

10

Sports Music News Personal Websites Travel Weird

Webby Judges

Internet professionals who work with and on the internet: new media journalists, editors, web developers, and other Internet professionals have clearly demonstrable familiarity with the category which they review

11

3 Stage Judging Process

Review Stage: From 3000 to 400 sites

3 judges rate each site on 6 criteria, and cast a vote if it will go to the next stage

Nominating Stage: From 400 to 135 sites

3 judges rate each site on 6 criteria, and cast a vote if it will go to the next stage

Final Stage: From 135 to 27 sites Judges cast vote for best site
12

Criteria for Judging

6 criteria

Content Structure & navigation Visual design Functionality Interactivity Overall experience

Scale: 1-10 (highest) Nearly normally distributed

13

Content
is the information provided on the site. Good content is engaging, relevant, appropriate for the audience-you can tell it's been developed for the Web because it's clear and concise and it works in the medium
14

Visual Design
is the appearance of the site. Good visual design is high quality, appropriate, and relevant for the audience and the message it is supporting

15

Interactivity
is the way a site allows a user to do something. Good interactivity is more than sound effects, and a Flash animation. It allows the user to give and receive. Its input/output in searches, chat rooms, ecommerce etc.

16

Can overall rating be predicted by specific criteria?

Statistical Technique: Regression analysis Question: What % variance is explained by 5 criteria

Percentage variance explained = 89%

Can votes be predicted by specific criteria?

Statistical Technique: Discriminant analysis Question: Can we predict the votes from the 5 specific criteria?

Classification Accuracy for Sites = 91%

Review Stage: Which criteria contribute most to overall rating?

Figure 2a. Review Stage Contribution of Specific Criteria to Overall Site Rating
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3

Content

Navigation

VisualDesign

Interactivity

Functionality

Nominating Stage Analysis

6 criteria

Content, Structure & Navigation, Visual Design, Functionality & Interactivity Overall experience

400 sites 3 judges rated each site

Nominating Stage: Top sites for each category

300

Overall Rating

200

100

Mean = 7.6 SD = 1.66

0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0

overall

Which criteria contribute to overall rating at Nominating Stage?

77% variance explained in overall rating
Contribution of Criteria (Correlation) Unique Contribution of Criteria (Partial Correlation)

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

Content

Navigation

VisualDesign

Interactivity

Functionality

Summary of Study I Findings

The specific ratings do explain overall experience. The best predictor of overall score is content. The second best predictor is interactivity. The worst predictor is visual design

23

Are there differences between categories?

Arts Activism Fashion Health News Sports Music News Personal Websites Travel

24

Art
Arts: Contribution of criteria to overall rating Variance explained = 93% 1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5

Content

Navigation VisualDesign Interactivity Functionality

Commerce Sites
Commerce Sites: Contribution of criteria to overall rating Variance explained = 87% 1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5

Content

Navigation VisualDesign Interactivity Functionality

Radio Sites
Radio Sites: Contribution of criteria to overall rating Variance explained = 90% 1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5

Content

Navigation VisualDesign Interactivity Functionality

Conclusions: Study I

The importance of criteria varies by category. Content is by far the best predictor of overall site experience. Interactivity comes next. Visual Design does not have as much predictive power except in specific categories

28

Study II

An empirical bottom-up approach to developing design guidelines

Challenge: How to go use Webby criteria to inform web page design?

Answer: Identify quantitative measures that characterize pages

29

Quantitative Measures
Page Composition
words, links, images,

Page Formatting
fonts, lists, colors,

Overall Characteristics
information & layout quality

30

Quantitative page measures

Word Count Body Text % Emphasized Body Text % Text Cluster Count Link Count Page Size Graphic % Color Count Font Count

Quantitative Measures:
Word Count

32

Quantitative Measures:
Body Text %

33

Quantitative Measures:
Emphasized Body Text %

34

Quantitative Measures:
Text Positioning Count

35

Quantitative Measures:
Text Cluster Count

36

Quantitative Measures:
Link Count

37

Quantitative Measures:
Page Size (Bytes)

38

Quantitative Measures:
Graphic %

39

Quantitative Measures:
Graphic Count

40

Study Design
Quantitative Page Metrics Webby Ratings Model Accuracy Within Across Categories Categories

Word Count, Body Text %, Text Cluster Count, Link Count etc.

Highly Rated Sites Top 33%

67%

76%

Low Rated Sites Bottom 33%

63%

83%

Classification Accuracy

Comparing Top vs. bottom Accuracy higher for within categories

N Classification Accuracy Top Bottom 67% 63% 83% 92% 76% 73% 77% 93% 93% 87% 42% 76% 86% 75% 76% 83%

Overall Within Community Categories Education Finance Health Living Services Cat. Avg.

1286 305 368 142 165 106 208

Which page metrics predict site quality?

All metrics played a role

However their role differed for various categories of pages (small, medium & large) Across all pages in the sample

Summary

Good pages had significantly smaller graphics percentage Good pages had less emphasized body text Good pages had more colors (on text)

Role of Metrics for Medium Pages

(230 words on average)

Good medium pages

Emphasize less of the body text Appear to organize text into clusters (e.g., lists and shaded table areas) Use colors to distinguish headings from body text

Suggests that these pages

Are easier to scan

44

Low Rated Page

No Text Clustering No Selective Highlighting

45

High Rated Page

Selective Highlighting

Text Clustering

46

Why does this approach work?

Superficial page metrics reflect deeper aspects of information architecture, interactivity etc.

47

Possible Uses

A grammar checker to assess guideline conformance

Imperfect Only suggestions not dogma

Automatic template suggestions Automatic comparison to highly usable pages/sites

48

Current Design Analysis Tools

Some tools report on easyto-measure attributes

Compare measures to thresholds Guideline conformance

49

Comparing a Design to Validated Good Designs

Web Site Design Profiles Comparable Designs

Analysis Tool

Favorite Designs
Prediction Similarities Differences Suggestions

50

Future work

Distinguish according to page role

Home page vs. content vs. index

Better metrics

More aspects of info, navigation, and graphic design

Site level as well as page level Category-based profiles

Use clustering to create profiles of good and poor sites These can be used to suggest alternative designs

51

Conclusions: Study II

Automated tools should help close the Web Usability Gap We have a foundation for a new methodology

Empirical, bottom up
Empirical validation of design guidelines Can build profiles of good vs. poor sites

We can empirically distinguish good pages

Eventually build tools to help users assess designs

52

More Information

http://webtango.berkeley.edu

53