Approximate methods for scalable data mining
Download as ppt, pdf4 likes2,661 views
The document discusses approximate methods for scalable data mining, emphasizing the trade-off between accuracy and scalability using probabilistic data structures like sketches and signatures. It outlines techniques such as cardinality estimation, hyperloglog algorithms, bloom filters, and count-min sketches, which allow efficient querying and data representation with reduced resource requirements. The document also provides resources for further reading and implementation examples in various programming environments.
1 of 15
Downloaded 24 times
![Approximate methods for scalable data mining l 24/04/134
What are approximate methods?
Trading accuracy for scalability
• Represent characteristics or summary of data
• Use much less space than full dataset (generally via hashing tricks)
– Can alleviate disk, memory, network bottlenecks
• Generally incur more CPU load than exact methods
– This may not be true in a distributed system, overall
○ [de]serialization for example
– Many data-centric systems have CPU to spare anyway](https://image.slidesharecdn.com/approximatemethodsshortversion-130424144248-phpapp01/85/Approximate-methods-for-scalable-data-mining-4-320.jpg)
Ad
Recommended
Approximate methods for scalable data mining (long version)
Approximate methods for scalable data mining (long version)Andrew Clegg This document provides an overview of approximate methods for scalable data mining. It discusses how approximate methods trade accuracy for scalability by using probabilistic data structures. Specific approximate methods covered include Bloom filters for set membership, probabilistic counting algorithms for cardinality estimation, count-min sketches for frequency estimation, and locality-sensitive hashing for similarity search. The document explains the algorithms and properties of these approximate methods.
Latency SLOs done right
Latency SLOs done rightFred Moyer This document discusses best practices for defining and measuring latency service level objectives (SLOs). It recommends computing SLOs directly from raw log data using histograms, which allow arbitrary percentiles to be derived and are better than averaging sample percentiles. Histograms can also be aggregated over time and used to count the number of requests above a latency threshold regardless of what the threshold was set to initially. Common histogram implementations like HDR-Histogram and t-digest are suggested.
OSTU - Sake Blok on TShark Statistics
OSTU - Sake Blok on TShark StatisticsDenny K Miu This document provides an overview of a training session on the use of tshark for displaying network statistics, focusing on its -z option for statistical analysis. It covers displaying protocol hierarchies, conversations, and IO statistics through various examples. The content is targeted at users wanting to enhance their network analysis skills using tshark.
Probabilistic Data Structures and Approximate Solutions Oleksandr Pryymak
Probabilistic Data Structures and Approximate Solutions Oleksandr PryymakPyData The document discusses probabilistic data structures and approximate solutions, highlighting their advantages such as reduced space requirements and scalability, while acknowledging potential worst-case performance issues. It covers various topics including hash functions, bloom filters, cardinality estimation, and machine learning techniques, supported by code examples. The overall emphasis is on understanding the trade-offs involved and effectively controlling errors in these approaches.
Probabilistic Data Structures and Approximate Solutions
Probabilistic Data Structures and Approximate SolutionsOleksandr Pryymak Probabilistic and approximate data structures can provide scalable solutions when exact answers are not required. They trade accuracy for speed and efficiency. Approaches like sampling, hashing, cardinality estimation, and probabilistic databases allow analyzing large datasets while controlling error rates. Example techniques discussed include Bloom filters, locality-sensitive hashing, count-min sketches, HyperLogLog, and feature hashing for machine learning. The talk provided code examples and comparisons of these probabilistic methods.
Probabilistic data structure
Probabilistic data structureThinh Dang Four main types of probabilistic data structures are described: membership, cardinality, frequency, and similarity. Bloom filters and cuckoo filters are discussed as membership data structures that can tell if an element is definitely not or may be in a set. Cardinality structures like HyperLogLog are able to estimate large cardinalities with small error rates. Count-Min Sketch is presented as a frequency data structure. MinHash and locality sensitive hashing are covered as similarity data structures that can efficiently find similar documents in large datasets.
Approximation Data Structures for Streaming Applications
Approximation Data Structures for Streaming ApplicationsDebasish Ghosh The document discusses approximation data structures for handling streaming data applications, emphasizing the challenges posed by unbounded data streams that require low-latency processing. It explores various algorithms and structures, such as count-min sketch and Bloom filters, to perform tasks like distinct element counting, frequency estimation, and heavy hitter identification with limited memory usage. The document also highlights the necessity of randomized and approximate methods to achieve efficient data processing under the constraints of space and accuracy.
An introduction to probabilistic data structures
An introduction to probabilistic data structuresMiguel Ping This document provides an overview of probabilistic data structures. It describes how they trade accuracy for speed or space by providing approximate results. Specific probabilistic data structures covered include Bloom filters, count-min sketch, t-digest, and hyperloglog. Bloom filters are used for set membership tests and can return false positives but not false negatives. Count-min sketch counts frequencies/popular items with a two-dimensional array. T-digest approximates quantiles through a sparse representation of the cumulative distribution function. Hyperloglog estimates cardinality by observing leading zeros in binary representations.
HyperLogLog and friends
HyperLogLog and friendsSimon Lia-Jonassen The document discusses advanced techniques for counting distinct items and frequent items using probabilistic data structures like HyperLogLog and its variants, which are efficient in terms of memory and parallelization. It includes comparisons of different methods and their performance metrics, noting applications across various frameworks such as Postgres and Hadoop. Additional resources and references, including code repositories and relevant literature, are provided for further exploration of these techniques.
Tech talk Probabilistic Data Structure
Tech talk Probabilistic Data StructureRishabh Dugar This document provides an overview of probabilistic data structures including Bloom filters, Cuckoo filters, Count-Min sketch, majority algorithm, linear counting, LogLog, HyperLogLog, locality sensitive hashing, minhash, and simhash. It discusses how Bloom filters work using hash functions to insert elements into a bit array and can tell if an element is likely present or definitely not present. It also explains how Count-Min sketch tracks frequencies in a stream using hash functions and finding the minimum value in each hash table cell. Finally, it summarizes how HyperLogLog estimates cardinality by tracking the maximum number of leading zeros when hashing random numbers.
Building graphs to discover information by David Martínez at Big Data Spain 2015
Building graphs to discover information by David Martínez at Big Data Spain 2015Big Data Spain Graphs can be built from raw data to discover information by representing relationships between data points as graph connections. Techniques like locality sensitive hashing can be used to efficiently construct graphs from high-dimensional data by mapping similar points to the same "buckets". Once a graph is built, algorithms can find structure like connected components, detect anomalies using local outlier factor, perform clustering, and make inferences about unlabeled nodes. Building graphs is a powerful approach for transforming raw data into useful information through network analysis and machine learning on graphs.
Count-Distinct Problem
Count-Distinct ProblemKai Zhang The document discusses count-distinct algorithms for estimating the cardinality of large data streams. It provides an overview of the history of count-distinct algorithms, from early linear counting approaches to modern algorithms like LogLog counting and HyperLogLog counting. The document then describes the basic ideas, algorithms, and implementations of LogLog counting and HyperLogLog counting. It analyzes the performance of these algorithms and discusses open issues like how to handle small and large cardinalities more accurately.
Beyond PFCount: Shrif Nada
Beyond PFCount: Shrif NadaRedis Labs This document provides an overview and summary of the HyperMinHash sketch, which is a probabilistic data structure that can estimate the size of a dataset and the size of unions between datasets using only logarithmic space. It discusses how counting problems like "how many users like cats or dogs" can be solved efficiently using sketching techniques instead of needing to process all the raw data. The HyperMinHash sketch works by hashing elements and tracking the largest number of leading zeros observed in each "bucket" to estimate sizes, and sketches can be merged to estimate unions between sets. It enables queries like set operations and similarity to be performed quickly and cheaply on large datasets.
2013 open analytics_countingv3
2013 open analytics_countingv3Open Analytics This document discusses cardinality estimation techniques for very large data sets. It begins by outlining goals for counting solutions such as supporting high throughput data streams, estimating cardinality within known error thresholds for large data sets up to 1 billion or 1 trillion elements, and supporting set operations. It then discusses naive solutions and their limitations before introducing intuitions and techniques like applying multiple hash functions, stochastic averaging, and the HyperLogLog algorithm. HyperLogLog uses a single hash function to map values to substreams and tracks the maximum value in each to estimate cardinality within a typical 1.04/m error. The document concludes by discussing related probabilistic data structures and references.
Probabilistic data structures. Part 2. Cardinality
Probabilistic data structures. Part 2. CardinalityAndrii Gakhov This document discusses various probabilistic data structures for estimating cardinality, including linear counting, HyperLogLog, and HyperLogLog++. It explains the algorithms, advantages, and applications of each method, highlighting memory efficiency and accuracy for large datasets. Additionally, the document presents examples and compares the performance of these techniques in different contexts.
Distributed algorithms for big data @ GeeCon
Distributed algorithms for big data @ GeeConDuyhai Doan This document discusses distributed algorithms for big data. It begins with an overview of HyperLogLog for estimating cardinality and counting distinct elements in a large data set. It then explains how HyperLogLog works by using a hash function to distribute the data across buckets and applying the LogLog algorithm to each bucket before taking the harmonic mean. The document also covers Paxos for distributed consensus, explaining the phases of prepare, promise, accept and learn to reach agreement in the presence of failures.
Flink Forward Berlin 2017: David Rodriguez - The Approximate Filter, Join, an...
Flink Forward Berlin 2017: David Rodriguez - The Approximate Filter, Join, an...Flink Forward The document by Cisco Systems discusses techniques for filtering, joining, and grouping data, focusing on applications like clickstream analysis, CPU usage tracking, and signal clustering using methods like Locality Sensitive Hashing (LSH). It provides examples and challenges related to data processing and emphasizes the importance of unsupervised methods in detecting patterns and anomalies, such as botnet detection. The content involves code snippets and algorithms illustrating these concepts, addressing practical problems in data science and network security.
Scribed lec8
Scribed lec8Praveen Kumar This document summarizes key points from a lecture on algorithms for analyzing graphs and finding similar data points:
1) It describes a naive algorithm for tracking connected components in an undirected graph and updates it runs in O(M+N log N) time, where M is edges and N is nodes.
2) It then introduces the more efficient Union-Find algorithm that represents each component as a tree and finds components in O(log N) time per update.
3) It concludes by explaining Locality-Sensitive Hashing, a technique to solve near neighbor problems by hash families that map similar points to the same buckets, allowing neighbor queries in sublinear time.
HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardin...
HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardin...Sunny Kr This document details improvements to the HyperLogLog algorithm for cardinality estimation, enhancing its accuracy and reducing memory requirements for large datasets. The authors present empirical evaluations and discuss the applicability of these improvements within Google's data analysis systems. Key enhancements include the use of a 64-bit hash function, corrections for small and large cardinality ranges, and overall efficiency in cardinality estimation tasks.
Data streaming algorithms
Data streaming algorithmsSandeep Joshi 1) The document discusses algorithms for computing statistics like minimum, maximum, average over data streams using limited memory in a single pass. It covers algorithms for computing cardinality, heavy hitters, order statistics and histograms.
2) Cardinality can be estimated using the Flajolet-Martin algorithm which tracks the position of the rightmost zero bit in a bitmap. Heavy hitters can be found using the Count-Min sketch. Order statistics like the median can be approximated using the Frugal and T-Digest algorithms. Wavelet-based approaches can be used to compute histograms over data streams.
3) The document provides high-level explanations of these streaming algorithms along with references for further reading, but does not
Probabilistic data structures
Probabilistic data structuresshrinivasvasala The document discusses probabilistic data structures including hyperloglog, bloom filters, and count-min sketches, which offer efficient approximate solutions for analyzing large data sets. Hyperloglog can estimate cardinalities with high accuracy using minimal memory, bloom filters help test set membership with an error rate, and count-min sketches allow frequency estimation of items in a collection. It also provides references for further reading on these topics.
Hyper loglog
Hyper loglogEugene Kostieiev The document discusses the HyperLogLog algorithm for estimating the number of distinct elements in large datasets, emphasizing its application in areas like Google search queries and genomic analysis. It highlights the importance of probabilistic counting methods and provides an overview of the algorithm's structure, advantages, and associated complexities. Additionally, it covers practical implementations and comparisons in various technologies like Elasticsearch and Redis.
2013 py con awesome big data algorithms
2013 py con awesome big data algorithmsc.titus.brown This document provides an overview of algorithms for analyzing large datasets, referred to as "big data". It discusses skip lists, HyperLogLog counting, and Bloom filters as examples of probabilistic data structures that can be used for problems involving big data. These algorithms provide approximate answers but are more scalable and memory efficient than exact algorithms. The document also describes applications of these algorithms to analyzing shotgun DNA sequencing data from metagenomics studies.
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...Andrii Gakhov Andrii Gakhov's presentation at EuroSciPy 2019 discusses the use of probabilistic data structures and algorithms (PDSA) in managing big data applications, focusing on efficient counting and frequency estimation in data streams. He highlights challenges in bioinformatics, such as counting unique k-mers in DNA sequences, and presents solutions like count-min sketches and hyperloglog for scalable frequency estimation and counting unique visitors. Gakhov emphasizes the need for new algorithms and technologies in dealing with the complexities of big data rather than relying solely on increased hardware resources.
Counting (Using Computer)
Counting (Using Computer)roshmat The document discusses methods for counting various elements in computer systems, focusing on memory-efficient techniques like probabilistic counting and estimators. It explores the limitations of fixed-size counters and introduces concepts like biased coins and hash functions for approximating counts and unique elements. Additionally, it outlines the trade-offs between accuracy and space efficiency in counting algorithms.
2013 open analytics_countingv3
2013 open analytics_countingv3abramsm This document discusses cardinality estimation techniques for very large data sets. It introduces HyperLogLog (HLL), an algorithm for distinct count estimation that uses stochastic averaging and hash-based binning to estimate the cardinality of data sets containing up to billions of elements using only 1.5KB of memory. The document explains how HLL works, including how values are added and cardinality is estimated from the HLL data structure. It also discusses extensions like HLL++ and related probabilistic data structures.
Large-scale real-time analytics for everyone
Large-scale real-time analytics for everyonePavel Kalaidin This document discusses probabilistic algorithms and data structures called sketches that can be used for large-scale real-time analytics using limited memory. It introduces HyperLogLog, a sketch that can accurately estimate the number of unique elements in a data stream using very little memory. It then discusses extensions like HyperLogLog++, sliding window HyperLogLog for tracking churn, and sketches for other tasks like percentiles, heavy hitters, and cardinality of set intersections. The document emphasizes that sketches provide approximate but memory-efficient solutions for analytics problems on streaming data.
Probabilistic data structures
Probabilistic data structuresYoav chernobroda This document discusses probabilistic data structures that are useful for analyzing big data streams. It covers techniques like sampling, bloom filters, cuckoo filters, count-min sketch, t-digest, and hyperloglog that allow estimating statistics of large datasets in a memory and computationally efficient manner. These probabilistic structures trade exact answers for performance and can estimate things like frequencies, quantiles, set membership, and cardinality in sub-linear time and space. Real-world examples of applications in domains like analytics, anomaly detection, and distributed systems are also presented.
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdfAmirStern2 מרכז עיבודים תעשייתי בעל 3/4/5 צירים, עד 22 החלפות כלים עם כל אפשרויות העיבוד הדרושות. בעל שטח עבודה גדול ומחשב נוח וקל להפעלה בשפה העברית/רוסית/אנגלית/ספרדית/ערבית ועוד..
מסוגל לבצע פעולות עיבוד שונות המתאימות לענפים שונים: קידוח אנכי, אופקי, ניסור, וכרסום אנכי.
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to productionMichele Leroux Bustamante We live in an ever evolving landscape for cyber threats creating security risk for your production systems. Mitigating these risks requires participation throughout all stages from development through production delivery - and by every role including architects, developers QA and DevOps engineers, product owners and leadership. No one is excused! This session will cover examples of common mistakes or missed opportunities that can lead to vulnerabilities in production - and ways to do better throughout the development lifecycle.
More Related Content
Similar to Approximate methods for scalable data mining (20)
HyperLogLog and friends
HyperLogLog and friendsSimon Lia-Jonassen The document discusses advanced techniques for counting distinct items and frequent items using probabilistic data structures like HyperLogLog and its variants, which are efficient in terms of memory and parallelization. It includes comparisons of different methods and their performance metrics, noting applications across various frameworks such as Postgres and Hadoop. Additional resources and references, including code repositories and relevant literature, are provided for further exploration of these techniques.
Tech talk Probabilistic Data Structure
Tech talk Probabilistic Data StructureRishabh Dugar This document provides an overview of probabilistic data structures including Bloom filters, Cuckoo filters, Count-Min sketch, majority algorithm, linear counting, LogLog, HyperLogLog, locality sensitive hashing, minhash, and simhash. It discusses how Bloom filters work using hash functions to insert elements into a bit array and can tell if an element is likely present or definitely not present. It also explains how Count-Min sketch tracks frequencies in a stream using hash functions and finding the minimum value in each hash table cell. Finally, it summarizes how HyperLogLog estimates cardinality by tracking the maximum number of leading zeros when hashing random numbers.
Building graphs to discover information by David Martínez at Big Data Spain 2015
Building graphs to discover information by David Martínez at Big Data Spain 2015Big Data Spain Graphs can be built from raw data to discover information by representing relationships between data points as graph connections. Techniques like locality sensitive hashing can be used to efficiently construct graphs from high-dimensional data by mapping similar points to the same "buckets". Once a graph is built, algorithms can find structure like connected components, detect anomalies using local outlier factor, perform clustering, and make inferences about unlabeled nodes. Building graphs is a powerful approach for transforming raw data into useful information through network analysis and machine learning on graphs.
Count-Distinct Problem
Count-Distinct ProblemKai Zhang The document discusses count-distinct algorithms for estimating the cardinality of large data streams. It provides an overview of the history of count-distinct algorithms, from early linear counting approaches to modern algorithms like LogLog counting and HyperLogLog counting. The document then describes the basic ideas, algorithms, and implementations of LogLog counting and HyperLogLog counting. It analyzes the performance of these algorithms and discusses open issues like how to handle small and large cardinalities more accurately.
Beyond PFCount: Shrif Nada
Beyond PFCount: Shrif NadaRedis Labs This document provides an overview and summary of the HyperMinHash sketch, which is a probabilistic data structure that can estimate the size of a dataset and the size of unions between datasets using only logarithmic space. It discusses how counting problems like "how many users like cats or dogs" can be solved efficiently using sketching techniques instead of needing to process all the raw data. The HyperMinHash sketch works by hashing elements and tracking the largest number of leading zeros observed in each "bucket" to estimate sizes, and sketches can be merged to estimate unions between sets. It enables queries like set operations and similarity to be performed quickly and cheaply on large datasets.
2013 open analytics_countingv3
2013 open analytics_countingv3Open Analytics This document discusses cardinality estimation techniques for very large data sets. It begins by outlining goals for counting solutions such as supporting high throughput data streams, estimating cardinality within known error thresholds for large data sets up to 1 billion or 1 trillion elements, and supporting set operations. It then discusses naive solutions and their limitations before introducing intuitions and techniques like applying multiple hash functions, stochastic averaging, and the HyperLogLog algorithm. HyperLogLog uses a single hash function to map values to substreams and tracks the maximum value in each to estimate cardinality within a typical 1.04/m error. The document concludes by discussing related probabilistic data structures and references.
Probabilistic data structures. Part 2. Cardinality
Probabilistic data structures. Part 2. CardinalityAndrii Gakhov This document discusses various probabilistic data structures for estimating cardinality, including linear counting, HyperLogLog, and HyperLogLog++. It explains the algorithms, advantages, and applications of each method, highlighting memory efficiency and accuracy for large datasets. Additionally, the document presents examples and compares the performance of these techniques in different contexts.
Distributed algorithms for big data @ GeeCon
Distributed algorithms for big data @ GeeConDuyhai Doan This document discusses distributed algorithms for big data. It begins with an overview of HyperLogLog for estimating cardinality and counting distinct elements in a large data set. It then explains how HyperLogLog works by using a hash function to distribute the data across buckets and applying the LogLog algorithm to each bucket before taking the harmonic mean. The document also covers Paxos for distributed consensus, explaining the phases of prepare, promise, accept and learn to reach agreement in the presence of failures.
Flink Forward Berlin 2017: David Rodriguez - The Approximate Filter, Join, an...
Flink Forward Berlin 2017: David Rodriguez - The Approximate Filter, Join, an...Flink Forward The document by Cisco Systems discusses techniques for filtering, joining, and grouping data, focusing on applications like clickstream analysis, CPU usage tracking, and signal clustering using methods like Locality Sensitive Hashing (LSH). It provides examples and challenges related to data processing and emphasizes the importance of unsupervised methods in detecting patterns and anomalies, such as botnet detection. The content involves code snippets and algorithms illustrating these concepts, addressing practical problems in data science and network security.
Scribed lec8
Scribed lec8Praveen Kumar This document summarizes key points from a lecture on algorithms for analyzing graphs and finding similar data points:
1) It describes a naive algorithm for tracking connected components in an undirected graph and updates it runs in O(M+N log N) time, where M is edges and N is nodes.
2) It then introduces the more efficient Union-Find algorithm that represents each component as a tree and finds components in O(log N) time per update.
3) It concludes by explaining Locality-Sensitive Hashing, a technique to solve near neighbor problems by hash families that map similar points to the same buckets, allowing neighbor queries in sublinear time.
HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardin...
HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardin...Sunny Kr This document details improvements to the HyperLogLog algorithm for cardinality estimation, enhancing its accuracy and reducing memory requirements for large datasets. The authors present empirical evaluations and discuss the applicability of these improvements within Google's data analysis systems. Key enhancements include the use of a 64-bit hash function, corrections for small and large cardinality ranges, and overall efficiency in cardinality estimation tasks.
Data streaming algorithms
Data streaming algorithmsSandeep Joshi 1) The document discusses algorithms for computing statistics like minimum, maximum, average over data streams using limited memory in a single pass. It covers algorithms for computing cardinality, heavy hitters, order statistics and histograms.
2) Cardinality can be estimated using the Flajolet-Martin algorithm which tracks the position of the rightmost zero bit in a bitmap. Heavy hitters can be found using the Count-Min sketch. Order statistics like the median can be approximated using the Frugal and T-Digest algorithms. Wavelet-based approaches can be used to compute histograms over data streams.
3) The document provides high-level explanations of these streaming algorithms along with references for further reading, but does not
Probabilistic data structures
Probabilistic data structuresshrinivasvasala The document discusses probabilistic data structures including hyperloglog, bloom filters, and count-min sketches, which offer efficient approximate solutions for analyzing large data sets. Hyperloglog can estimate cardinalities with high accuracy using minimal memory, bloom filters help test set membership with an error rate, and count-min sketches allow frequency estimation of items in a collection. It also provides references for further reading on these topics.
Hyper loglog
Hyper loglogEugene Kostieiev The document discusses the HyperLogLog algorithm for estimating the number of distinct elements in large datasets, emphasizing its application in areas like Google search queries and genomic analysis. It highlights the importance of probabilistic counting methods and provides an overview of the algorithm's structure, advantages, and associated complexities. Additionally, it covers practical implementations and comparisons in various technologies like Elasticsearch and Redis.
2013 py con awesome big data algorithms
2013 py con awesome big data algorithmsc.titus.brown This document provides an overview of algorithms for analyzing large datasets, referred to as "big data". It discusses skip lists, HyperLogLog counting, and Bloom filters as examples of probabilistic data structures that can be used for problems involving big data. These algorithms provide approximate answers but are more scalable and memory efficient than exact algorithms. The document also describes applications of these algorithms to analyzing shotgun DNA sequencing data from metagenomics studies.
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...
Exceeding Classical: Probabilistic Data Structures in Data Intensive Applicat...Andrii Gakhov Andrii Gakhov's presentation at EuroSciPy 2019 discusses the use of probabilistic data structures and algorithms (PDSA) in managing big data applications, focusing on efficient counting and frequency estimation in data streams. He highlights challenges in bioinformatics, such as counting unique k-mers in DNA sequences, and presents solutions like count-min sketches and hyperloglog for scalable frequency estimation and counting unique visitors. Gakhov emphasizes the need for new algorithms and technologies in dealing with the complexities of big data rather than relying solely on increased hardware resources.
Counting (Using Computer)
Counting (Using Computer)roshmat The document discusses methods for counting various elements in computer systems, focusing on memory-efficient techniques like probabilistic counting and estimators. It explores the limitations of fixed-size counters and introduces concepts like biased coins and hash functions for approximating counts and unique elements. Additionally, it outlines the trade-offs between accuracy and space efficiency in counting algorithms.
2013 open analytics_countingv3
2013 open analytics_countingv3abramsm This document discusses cardinality estimation techniques for very large data sets. It introduces HyperLogLog (HLL), an algorithm for distinct count estimation that uses stochastic averaging and hash-based binning to estimate the cardinality of data sets containing up to billions of elements using only 1.5KB of memory. The document explains how HLL works, including how values are added and cardinality is estimated from the HLL data structure. It also discusses extensions like HLL++ and related probabilistic data structures.
Large-scale real-time analytics for everyone
Large-scale real-time analytics for everyonePavel Kalaidin This document discusses probabilistic algorithms and data structures called sketches that can be used for large-scale real-time analytics using limited memory. It introduces HyperLogLog, a sketch that can accurately estimate the number of unique elements in a data stream using very little memory. It then discusses extensions like HyperLogLog++, sliding window HyperLogLog for tracking churn, and sketches for other tasks like percentiles, heavy hitters, and cardinality of set intersections. The document emphasizes that sketches provide approximate but memory-efficient solutions for analytics problems on streaming data.
Probabilistic data structures
Probabilistic data structuresYoav chernobroda This document discusses probabilistic data structures that are useful for analyzing big data streams. It covers techniques like sampling, bloom filters, cuckoo filters, count-min sketch, t-digest, and hyperloglog that allow estimating statistics of large datasets in a memory and computationally efficient manner. These probabilistic structures trade exact answers for performance and can estimate things like frequencies, quantiles, set membership, and cardinality in sub-linear time and space. Real-world examples of applications in domains like analytics, anomaly detection, and distributed systems are also presented.
Recently uploaded (20)
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdfAmirStern2 מרכז עיבודים תעשייתי בעל 3/4/5 צירים, עד 22 החלפות כלים עם כל אפשרויות העיבוד הדרושות. בעל שטח עבודה גדול ומחשב נוח וקל להפעלה בשפה העברית/רוסית/אנגלית/ספרדית/ערבית ועוד..
מסוגל לבצע פעולות עיבוד שונות המתאימות לענפים שונים: קידוח אנכי, אופקי, ניסור, וכרסום אנכי.
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to productionMichele Leroux Bustamante We live in an ever evolving landscape for cyber threats creating security risk for your production systems. Mitigating these risks requires participation throughout all stages from development through production delivery - and by every role including architects, developers QA and DevOps engineers, product owners and leadership. No one is excused! This session will cover examples of common mistakes or missed opportunities that can lead to vulnerabilities in production - and ways to do better throughout the development lifecycle.
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FMESafe Software This session will cover a common use case for local and state/provincial governments who create and/or maintain their 9-1-1 addressing data, particularly address points and road centerlines. In this session, you'll learn how FME has helped Shelby County 9-1-1 (TN) automate the 9-1-1 addressing process; including automatically assigning attributes from disparate sources, on-the-fly QAQC of said data, and reporting. The FME logic that this presentation will cover includes: Table joins using attributes and geometry, Looping in custom transformers, Working with lists and Change detection.
Connecting Data and Intelligence: The Role of FME in Machine Learning
Connecting Data and Intelligence: The Role of FME in Machine LearningSafe Software In this presentation, we want to explore powerful data integration and preparation for Machine Learning. FME is known for its ability to manipulate and transform geospatial data, connecting diverse data sources into efficient and automated workflows. By integrating FME with Machine Learning techniques, it is possible to transform raw data into valuable insights faster and more accurately, enabling intelligent analysis and data-driven decision making.
Turning the Page – How AI is Exponentially Increasing Speed, Accuracy, and Ef...
Turning the Page – How AI is Exponentially Increasing Speed, Accuracy, and Ef...Impelsys Inc. Artificial Intelligence (AI) has become a game-changer in content creation, automating tasks that were once very time-consuming and labor-intensive. AI-powered tools are now capable of generating high-quality articles, blog posts, and even poetry by analyzing large datasets of text and producing human-like writing.
However, AI’s influence on content generation is not limited to text; it has also made advancements in multimedia content, such as image, video, and audio generation. AI-powered tools can now transform raw images and footage into visually stunning outputs, and are all set to have a profound impact on the publishing industry.
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Enhance GitHub Copilot using MCP - Enterprise version.pdfNilesh Gule Slide deck related to the GitHub Copilot Bootcamp in Melbourne on 17 June 2025
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Smarter Aviation Data Management: Lessons from Swedavia Airports and SwecoSafe Software Managing airport and airspace data is no small task, especially when you’re expected to deliver it in AIXM format without spending a fortune on specialized tools. But what if there was a smarter, more affordable way?
Join us for a behind-the-scenes look at how Sweco partnered with Swedavia, the Swedish airport operator, to solve this challenge using FME and Esri.
Learn how they built automated workflows to manage periodic updates, merge airspace data, and support data extracts – all while meeting strict government reporting requirements to the Civil Aviation Administration of Sweden.
Even better? Swedavia built custom services and applications that use the FME Flow REST API to trigger jobs and retrieve results – streamlining tasks like securing the quality of new surveyor data, creating permdelta and baseline representations in the AIS schema, and generating AIXM extracts from their AIS data.
To conclude, FME expert Dean Hintz will walk through a GeoBorders reading workflow and highlight recent enhancements to FME’s AIXM (Aeronautical Information Exchange Model) processing and interpretation capabilities.
Discover how airports like Swedavia are harnessing the power of FME to simplify aviation data management, and how you can too.
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptx
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptxFIDO Alliance FIDO Seminar: New Data: Passkey Adoption in the Workforce
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdfMuhammad Yuga Nugraha Explore the ease of managing Firecracker microVM with the firecracker-python. In this session, I will introduce the basics of Firecracker microVM and demonstrate how this custom SDK facilitates microVM operations easily. We will delve into the design and development process behind the SDK, providing a behind-the-scenes look at its creation and features. While traditional Firecracker SDKs were primarily available in Go, this module brings a simplicity of Python to the table.
Enabling BIM / GIS integrations with Other Systems with FME
Enabling BIM / GIS integrations with Other Systems with FMESafe Software Jacobs has successfully utilized FME to tackle the complexities of integrating diverse data sources in a confidential $1 billion campus improvement project. The project aimed to create a comprehensive digital twin by merging Building Information Modeling (BIM) data, Construction Operations Building Information Exchange (COBie) data, and various other data sources into a unified Geographic Information System (GIS) platform. The challenge lay in the disparate nature of these data sources, which were siloed and incompatible with each other, hindering efficient data management and decision-making processes.
To address this, Jacobs leveraged FME to automate the extraction, transformation, and loading (ETL) of data between ArcGIS Indoors and IBM Maximo. This process ensured accurate transfer of maintainable asset and work order data, creating a comprehensive 2D and 3D representation of the campus for Facility Management. FME's server capabilities enabled real-time updates and synchronization between ArcGIS Indoors and Maximo, facilitating automatic updates of asset information and work orders. Additionally, Survey123 forms allowed field personnel to capture and submit data directly from their mobile devices, triggering FME workflows via webhooks for real-time data updates. This seamless integration has significantly enhanced data management, improved decision-making processes, and ensured data consistency across the project lifecycle.
Raman Bhaumik - Passionate Tech Enthusiast
Raman Bhaumik - Passionate Tech EnthusiastRaman Bhaumik A Junior Software Developer with a flair for innovation, Raman Bhaumik excels in delivering scalable web solutions. With three years of experience and a solid foundation in Java, Python, JavaScript, and SQL, she has streamlined task tracking by 20% and improved application stability.
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/from-enterprise-to-makers-driving-vision-ai-innovation-at-the-extreme-edge-a-presentation-from-sony-semiconductor-solutions/
Amir Servi, Edge Deep Learning Product Manager at Sony Semiconductor Solutions, presents the “From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge” tutorial at the May 2025 Embedded Vision Summit.
Sony’s unique integrated sensor-processor technology is enabling ultra-efficient intelligence directly at the image source, transforming vision AI for enterprises and developers alike. In this presentation, Servi showcases how the AITRIOS platform simplifies vision AI for enterprises with tools for large-scale deployments and model management.
Servi also highlights his company’s collaboration with Ultralytics and Raspberry Pi, which brings YOLO models to the developer community, empowering grassroots innovation. Whether you’re scaling vision AI for industry or experimenting with cutting-edge tools, this presentation will demonstrate how Sony is accelerating high-performance, energy-efficient vision AI for all.
Information Security Response Team Nepal_npCERT_Vice_President_Sudan_Jha.pdf
Information Security Response Team Nepal_npCERT_Vice_President_Sudan_Jha.pdfICT Frame Magazine Pvt. Ltd. Artificial Intelligence (AI) is rapidly changing the face of cybersecurity across the globe. In Nepal, the shift is already underway. Vice President of the Information Security Response Team Nepal (npCERT) and Information Security Consultant at One Cover Pvt. Ltd., Sudan Jha, recently presented an in-depth workshop on how AI can strengthen national security and digital defenses.
FIDO Seminar: Authentication for a Billion Consumers - Amazon.pptx
FIDO Seminar: Authentication for a Billion Consumers - Amazon.pptxFIDO Alliance FIDO Seminar: Authentication for a Billion Consumers - Amazon
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation Can We Use Rust to Develop Extensions for PostgreSQL?
(POSETTE: An Event for Postgres 2025)
June 11, 2025
Shinya Kato
NTT DATA Japan Corporation
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptxFIDO Alliance Securing Account Lifecycles in the Age of Deepfakes
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?Shashi Sathyanarayana, Ph.D This slide illustrates a side-by-side comparison between human-written, AI-written, and ambiguous content. It highlights subtle cues that help readers assess authenticity, raising essential questions about the future of communication, trust, and thought leadership in the age of generative AI.
FME for Distribution & Transmission Integrity Management Program (DIMP & TIMP)
FME for Distribution & Transmission Integrity Management Program (DIMP & TIMP)Safe Software Peoples Gas in Chicago, IL has changed to a new Distribution & Transmission Integrity Management Program (DIMP & TIMP) software provider in recent years. In order to successfully deploy the new software we have created a series of ETL processes using FME Form to transform our gas facility data to meet the required DIMP & TIMP data specifications. This presentation will provide an overview of how we used FME to transform data from ESRI’s Utility Network and several other internal and external sources to meet the strict data specifications for the DIMP and TIMP software solutions.
OWASP Barcelona 2025 Threat Model Library
OWASP Barcelona 2025 Threat Model LibraryPetraVukmirovic Threat Model Library Launch at OWASP Barcelona 2025
https://owasp.org/www-project-threat-model-library/
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to productionMichele Leroux Bustamante
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...Edge AI and Vision Alliance
Information Security Response Team Nepal_npCERT_Vice_President_Sudan_Jha.pdf
Information Security Response Team Nepal_npCERT_Vice_President_Sudan_Jha.pdfICT Frame Magazine Pvt. Ltd.
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation
Ad
Approximate methods for scalable data mining
- 2. Approximate methods for scalable data mining Andrew Clegg Data Analytics & Visualization Team Pearson Technology Twitter: @andrew_clegg
- 3. Approximate methods for scalable data mining l 24/04/133 What are approximate methods? Trading accuracy for scalability • Often use probabilistic data structures – a.k.a. sketches or signatures • Mostly stream-friendly – Allow you to query data you haven’t even kept! • Generally simple to parallelize • Predictable error rate (can be tuned)
- 4. Approximate methods for scalable data mining l 24/04/134 What are approximate methods? Trading accuracy for scalability • Represent characteristics or summary of data • Use much less space than full dataset (generally via hashing tricks) – Can alleviate disk, memory, network bottlenecks • Generally incur more CPU load than exact methods – This may not be true in a distributed system, overall ○ [de]serialization for example – Many data-centric systems have CPU to spare anyway
- 5. Approximate methods for scalable data mining l 24/04/135 Why approximate methods? A real-life example Icons from Dropline Neu! http://findicons.com/pack/1714/dropline_neu Counting unique terms in time buckets across ElasticSearch shards Cluster nodes Master node Unique terms per bucket per shard Globally unique terms per bucket Client Number of globally unique terms per bucket
- 6. Approximate methods for scalable data mining l 24/04/136 Why approximate methods? A real-life example Icons from Dropline Neu! http://findicons.com/pack/1714/dropline_neu But what if each bucket contains a LOT of terms? … and what if there are too many to fit in memory? Memory cost CPU cost to serialize Network transfer cost CPU cost to deserialize CPU & memory cost to merge & count sets
- 7. Approximate methods for scalable data mining l 24/04/137 Cardinality estimation Approximate distinct counts Intuitive explanation Long runs of trailing 0s in random bit strings are rare. But the more bit strings you look at, the more likely you are to see a long one. So “longest run of trailing 0s seen” can be used as an estimator of “number of unique bit strings seen”. 01110001 11101010 00100101 11001100 11110100 11101100 00010100 00000001 00000010 10001110 01110100 01101010 01111111 00100010 00110000 00001010 01000100 01111010 01011101 00000100
- 8. Approximate methods for scalable data mining l 24/04/138 Cardinality estimation Probabilistic counting: basic algorithm Counting the items • Let n = 0 • For each input item: – Hash item into bit string – Count trailing zeroes in bit string – If this count > n: ○ Let n = count Calculating the count • n = longest run of trailing 0s seen • Estimated cardinality (“count distinct”) = 2^n … that’s it! This is an estimate, but not a great one. But…
- 9. Approximate methods for scalable data mining l 24/04/139 HyperLogLog algorithm Billions of distinct values in 1.5KB of RAM with 2% relative error Image: http://www.aggregateknowledge.com/science/blog/hll.html Cool properties • Stream-friendly: no need to keep data • Error rates are predictable and tunable • Size and speed stay constant • Trivial to parallelize – Combine two HLL counters by taking the max of each register
- 10. Approximate methods for scalable data mining l 24/04/1310 Resources on cardinality estimation HyperLogLog paper: http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf Java implementations: https://github.com/clearspring/stream-lib/ Algebird implements HyperLogLog (and much more!) in Scalding: https://github.com/twitter/algebird Simmer wraps Algebird in Hadoop Streaming command line: https://github.com/avibryant/simmer Our ElasticSearch plugin: https://github.com/ptdavteam/elasticsearch-approx-plugin MetaMarkets blog: http://metamarkets.com/2012/fast-cheap-and-98-right-cardinality-estimation-for-big-data/ Aggregate Knowledge blog, including JavaScript implementation and D3 visualization: http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/
- 11. Approximate methods for scalable data mining l 24/04/1311 Bloom filters Set membership test with chance of false positives Image: http://en.wikipedia.org/wiki/Bloom_filter At least one 0 means w definitely isn’t in set. All 1s would mean w probably is in set. Hash each item n times ⇒ indices into bit field.
- 12. Approximate methods for scalable data mining l 24/04/1312 Count-min sketch Frequency histogram estimation with chance of over-counting A1 +1 +1 A2 +1 +1 A3 +2 “foo” h1 h2 h3 “bar” h1 h2 h3 More hashes / arrays ⇒ reduced chance of overcounting count(“foo”) = min(1, 1, 2) = 1
- 13. Approximate methods for scalable data mining l 24/04/1313 Random hyperplanes Locality-sensitive hashing for approximate nearest neighbours Hash(Item1) = 011 Hash(Item2) = 001 As the cosine distance decreases, the probability of a hash match increases Item1 h1 h2 h3 Item2 θ Bitwise hamming distance correlates with cosine distance
- 14. Approximate methods for scalable data mining l 24/04/1314 Feature hashing High-dimensional machine learning without feature dictionary “reduce” “the” “size” “of” “your” “feature” “vector” “with” “this” “one” “weird” “old” “trick” h(“reduce”) = 9 h(“the”) = 3 h(“size”) = 1 . . . +1 +1 +1 Effect of collisions on overall classification accuracy is surprisingly small! Multiple hashes, or 1-bit “sign hash”, can reduce collisions effects if necessary
- 15. Approximate methods for scalable data mining l 24/04/1315 Thanks for listening And some further reading… Great ebook available free from: http://infolab.stanford.edu/~ullman/mmds.html