Scribd
Upload
26 views

Web Mining Overview

This document provides an overview of the topics that will be covered in the CS 345A Data Mining course, with a focus on web mining. The key topics discussed include what web mining is, its differences from traditional data mining, analyzing the structure of the web as a graph, power laws and the long tail, extracting structured data from the web, web advertising, and systems issues involved in mining very large web-scale data. The course will provide hands-on experience with real-world web mining projects and infrastructure.

Uploaded by

Bhanu Praveen Kurakula
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
26 views

Web Mining Overview

This document provides an overview of the topics that will be covered in the CS 345A Data Mining course, with a focus on web mining. The key topics discussed include what web mining is, its differences from traditional data mining, analyzing the structure of the web as a graph, power laws and the long tail, extracting structured data from the web, web advertising, and systems issues involved in mining very large web-scale data. The course will provide hands-on experience with real-world web mining projects and infrastructure.

Uploaded by

Bhanu Praveen Kurakula
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 29

CS 345A

Data Mining
Lecture 1

Introduction to Web Mining


What is Web Mining?

Discovering useful information from


the World-Wide Web and its usage
patterns
Web Mining v. Data Mining
 Structure (or lack of it)
 Textual information and linkage structure
 Scale
 Data generated per day is comparable to
largest conventional data warehouses
 Speed
 Often need to react to evolving usage
patterns in real-time (e.g.,
merchandising)
Web Mining topics
 Web graph analysis
 Power Laws and The Long Tail
 Structured data extraction
 Web advertising
 Systems Issues
Web Mining topics
 Web graph analysis
 Power Laws and The Long Tail
 Structured data extraction
 Web advertising
 Systems Issues
Size of the Web
 Number of pages
 Technically, infinite
 Much duplication (30-40%)
 Best estimate of “unique” static HTML
pages comes from search engine claims
 Until last year, Google claimed 8 billion(?),
Yahoo claimed 20 billion
 Google recently announced that their index
contains 1 trillion pages
 How to explain the discrepancy?
The web as a graph
 Pages = nodes, hyperlinks = edges
 Ignore content
 Directed graph
 High linkage
 10-20 links/page on average
 Power-law degree distribution
Structure of Web graph
 Let’s take a closer look at structure
 Broder et al (2000) studied a crawl of
200M pages and other smaller crawls
 Bow-tie structure
 Not a “small world”
Bow-tie Structure

Source: Broder et al, 2000


What can the graph tell us?
 Distinguish “important” pages from
unimportant ones
 Page rank
 Discover communities of related
pages
 Hubs and Authorities
 Detect web spam
 Trust rank
Web Mining topics
 Web graph analysis
 Power Laws and The Long Tail
 Structured data extraction
 Web advertising
 Systems Issues
Power-law degree distribution

Source: Broder et al, 2000


Power-laws galore
 Structure
 In-degrees
 Out-degrees
 Number of pages per site
 Usage patterns
 Number of visitors
 Popularity e.g., products, movies, music
The Long Tail

Source: Chris Anderson (2004)


The Long Tail
 Shelf space is a scarce commodity for
traditional retailers
 Also: TV networks, movie theaters,…
 The web enables near-zero-cost
dissemination of information about
products
 More choice necessitates better filters
 Recommendation engines (e.g., Amazon)
 How Into Thin Air made Touching the Void a
bestseller
Web Mining topics
 Web graph analysis
 Power Laws and The Long Tail
 Structured data extraction
 Web advertising
 Systems Issues
Extracting Structured Data

http://www.simplyhired.com
Extracting structured data

http://www.fatlens.com
Web Mining topics
 Web graph analysis
 Power Laws and The Long Tail
 Structured data extraction
 Web advertising
 Systems Issues
Ads vs. search results
Ads vs. search results
 Search advertising is the revenue
model
 Multi-billion-dollar industry
 Advertisers pay for clicks on their ads
 Interesting problems
 What ads to show for a search?
 If I’m an advertiser, which search terms
should I bid on and how much to bid?
Web Mining topics
 Web graph analysis
 Power Laws and The Long Tail
 Structured data extraction
 Web advertising
 Systems Issues
Two Approaches to Analyzing
Data
 Machine Learning approach
 Emphasizes sophisticated algorithms
e.g., Support Vector Machines
 Data sets tend to be small, fit in memory
 Data Mining approach
 Emphasizes big data sets (e.g., in the
terabytes)
 Data cannot even fit on a single disk!
 Necessarily leads to simpler algorithms
Philosophy
 In many cases, adding more data
leads to better results that improving
algorithms
 Netflix
 Google search
 Google ads
 More on my blog:
Datawocky (datawocky.com)
Systems architecture

CPU
Machine Learning, Statistics

Memory

“Classical” Data Mining

Disk
Very Large-Scale Data Mining

CPU CPU CPU


…
Mem Mem Mem

Disk Disk Disk

Cluster of commodity nodes


Systems Issues
 Web data sets can be very large
 Tens to hundreds of terabytes
 Cannot mine on a single server!
 Need large farms of servers
 How to organize hardware/software
to mine multi-terabye data sets
 Without breaking the bank!
Web Mining topics
 Web graph analysis
 Power Laws and The Long Tail
 Structured data extraction
 Web advertising
 Systems Issues
Project
 Lots of interesting project ideas
 If you can’t think of one please come discuss
with us
 Infrastructure
 Aster Data cluster on Amazon EC2
 Supports both MapReduce and SQL
 Data
 Netflix
 ShareThis
 Google
 WebBase
 TREC

You might also like