![ Consider: |S| = m, |B| = n
Use k independent hash functions h1 ,…, hk
Initialization:
◦ Set B to all 0s
◦ Hash each element s S using each hash function hi, set
B[hi(s)] = 1 (for each i = 1,.., k)
Run-time:
◦ When a stream element with key x arrives
If B[hi(x)] = 1 for all i = 1,..., k then declare that x is in S
That is, x hashes to a bucket set to 1 for every hash
function hi(x)
Otherwise discard the element x](https://image.slidesharecdn.com/slideshare-miningdatastreams-200218115339/85/Mining-Data-Streams-15-320.jpg)
![ Datar-Gionis-Indyk-Motwani Algorithm
A bucket in the DGIM method is a record consisting
of:
1. The timestamp of its end [O(log N) bits]
2. The number of 1s between its beginning and end [O(log log
N) bits]
Constraint on buckets:
Number of 1s must be a power of 2
◦ That explains the O(log log N) in (2)](https://image.slidesharecdn.com/slideshare-miningdatastreams-200218115339/85/Mining-Data-Streams-23-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Mining Data Streams (20)
Ad
Recently uploaded (20)
Ad
Mining Data Streams
- 1. Suja A. Alex, Assistant Professor, Department of Information Technology, St. Xavier’s Catholic College of Engineering, Nagercoil
- 2. Introduction to stream concepts Stream Data Model Sampling data in a stream Filtering Stream Stream counting Decaying Window RTAP Real Time Data streaming tools and applications
- 3. Data Stream is continuously arriving data flow Examples: Data produced in dynamic environments like Power supply network traffic Stock exchange Video survelliance
- 4. Continuous flow of data Examples Network traffic Sensor data Call center records
- 5. Data Stream Management System (DSMS), has multiple data streams. A stream data query processing architecture includes three parts: end user, query processor, and scratch space
- 6. Transient streams (and persistent relations) Continuous queries Sequential access Unpredictable data arrival and characteristics Bounded main memory Types of Queries One time queries Continuous queries
- 7. Query Processing Continuous Query (CQ)Result Query Processing Main MemoryData Stream(s) Data Stream(s) Disk Main Memory SQL Query Result
- 10. Since we can not store the entire stream, one obvious approach is to store a sample. Types of sampling Reservoir Sampling Biased Reservoir Sampling Concise Sampling
- 11. Reservoir Sampling- A reservoir of size n is maintained The first t points are added (t+1) point is added with probability n/(t+1) Biased Reservoir Sampling – Bias function is used to regulate sampling from stream Concise Sampling – distinct values in stream are stored in main memory
- 12. Types of queries one wants on answer on a stream: ◦ Filtering a data stream Select elements with property x from the stream ◦ Counting distinct elements Number of distinct elements in the last k elements of the stream ◦ Estimating moments Estimate avg./std. dev. of last k elements ◦ Finding frequent elements
- 13. It decides whether an item from infinite stream must be stored Example- web browser storing the blacklist of dangerous URLs Each element of data stream is a tuple, given a list of keys S, Determine which tuples of stream are in S Obvious solution: Hash table But suppose we do not have enough memory to store all of S in a hash table
- 14. Example: Email spam filtering ◦ We know 1 billion “good” email addresses ◦ If an email comes from one of these, it is NOT spam Publish-subscribe systems ◦ You are collecting lots of messages (news articles) ◦ People express interest in certain sets of keywords ◦ Determine whether each message matches user’s interest
- 15. Consider: |S| = m, |B| = n Use k independent hash functions h1 ,…, hk Initialization: ◦ Set B to all 0s ◦ Hash each element s S using each hash function hi, set B[hi(s)] = 1 (for each i = 1,.., k) Run-time: ◦ When a stream element with key x arrives If B[hi(x)] = 1 for all i = 1,..., k then declare that x is in S That is, x hashes to a bucket set to 1 for every hash function hi(x) Otherwise discard the element x
- 16. Problem: ◦ Data stream consists of a universe of elements chosen from a set of size N ◦ Maintain a count of the number of distinct elements seen so far Solution: Maintain the set of elements seen so far ◦ That is, keep a hash table of all the distinct elements seen so far
- 17. How many different words are found among the Web pages being crawled at a site? ◦ Unusually low or high numbers could indicate artificial pages (spam?) How many different Web pages does each customer request in a week? How many distinct products have we sold in the last week?
- 18. Pick a hash function h that maps each of the N elements to at least log2 N bits For each stream element a, let r(a) be the number of trailing 0s in h(a) ◦ r(a) = position of first 1 counting from the right E.g., say h(a) = 12, then 12 is 1100 in binary, so r(a) = 2 Record R = the maximum r(a) seen ◦ R = maxa r(a), over all the items a seen so far Estimated number of distinct elements = 2R
- 19. Suppose a stream has elements chosen from a set A of N values Let mi be the number of times value i occurs in the stream The kth moment is Ai k im )(
- 20. 0thmoment = number of distinct elements ◦ The problem just considered 1st moment = count of the numbers of elements = length of the stream ◦ Easy to compute 2nd moment = surprise number S = a measure of how uneven the distribution is Alon Matias Szegedy (AMS) Algorithm
- 21.
- 22. q w e r t y u i o p a s d f g h j k l z x c v b n m q w e r t y u i o p a s d f g h j k l z x c v b n m q w e r t y u i o p a s d f g h j k l z x c v b n m q w e r t y u i o p a s d f g h j k l z x c v b n m Past Future N = 6
- 23. Datar-Gionis-Indyk-Motwani Algorithm A bucket in the DGIM method is a record consisting of: 1. The timestamp of its end [O(log N) bits] 2. The number of 1s between its beginning and end [O(log log N) bits] Constraint on buckets: Number of 1s must be a power of 2 ◦ That explains the O(log log N) in (2)
- 24. N 1 of size 2 2 of size 4 2 of size 8 At least 1 of size 16. Partially beyond window. 2 of size 1 1001010110001011010101010101011010101010101110101010111010100010110010 Properties we maintain: - Either one or two buckets with the same power-of-2 number of 1s - Buckets do not overlap in timestamps - Buckets are sorted by size
- 26. To estimate the number of 1s in the most recent N bits: 1. Sum the sizes of all buckets but the last (note “size” means the number of 1s in the bucket) 2. Add half the size of the last bucket Remember: We do not know how many 1s of the last bucket are still within the wanted window
- 27. Decay factor instead of sliding window technique Applications: The problem of Most Common Elements Example- movie tickets purchased all over the world Describing a Decaying Window
- 28. A real-time analytics platform enables organizations to make the most out of real-time data by helping them to extract the valuable information and trends from it. Real-time analytics is a process of delivering information about events as they occur Some Examples Financial Industry - Fraud Detection, Trading E-commerce - Recommendations Telecom Industry - Machine to Machine communication Supply Chain Management Business Activity Monitoring
- 29. On Demand Real Time Analytics Continuous Real Time Analytics
- 30. Delivering In-Memory Transaction Speed Quickly moving unneeded data to disk for long-term storage Distributing Data and Processing for speed Supporting continuous queries for real-time events Embedding Data into Apps or Apps into databases Additional Requirements-fault tolerance, low-latency
- 31. Processing in memory In-database analytics Data warehouse appliances In-memory analytics Massively Parallel Programming(MPP)
- 32. Real Time Sentiment Analysis (RTSA) Real Time Stock Prediction (RTSP)
- 33. Anand Rajaraman and Jeffrey David Ullman, "Mining of Massive Datasets", Cambridge University Press, 2012.