Introduction to Bloom Filters

Feb 26, 20200 likes70 views

This document provides an overview of Bloom filters, including what they are, why you would want to use one, practical examples of their use, and how they work. Bloom filters are a data structure that can be used to test if an element is present in a set. They have a tiny memory footprint compared to storing the full set, use constant memory, and allow extremely fast lookups. However, they can produce false positives. The document discusses examples of Bloom filters being used by companies like Akamai, Medium, and Cassandra to reduce storage usage and improve performance. It also explains how Bloom filters work under the hood using hash functions and bit arrays.

Hayden Marchant, Software Engineer
haydenm@wix.com
Bloom Filters

Agenda
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

A
Bloom-what?
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

Just a Data Structure
‘Is an element in a set
or not’

Why would I
want one?
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

Element
Key size:
# elements:
Memory
required:
100 bytes
10 million
100 * 10M
= 1GB
SIMPLE MATHS

Bloom Filters:
Tiny footprint
~ 10 bits per element

>98%
reduction
in space
Element
Key size:
# elements:
Memory
required:
10 bits
10 million
100M bits
~= 12MB
SIMPLE MATHS
(PART 2)

Footprints:
Traditional = 1GB
Bloom Filter <=12MB

Practical
Examples
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

Akamai
Prevent caching of
One-Hit-Wonders
75% of web requests are
only ever called once
- Called One-Hit-Wonders
Use Bloom Filters to prevent
one-hit-wonders to be stored
in Disk Cache
1. Saves disk-access for
these 75% of requests
2. reducing workload

Medium
Avoid recommending
articles a user has
read
- Add article recommended
to bloom filter -
$userid-$articleid
- Before recommending,
check in Bloom Filter if
(user,article) exists.
- If not, then recommend
and then add to bloom
filter

Cassandra
Reduce disk lookup for
non-existent rows in file
storage
- Shard responsible over
range of data
- Each shard stored in
immutable shard-files
- Create Bloom Filter in
server for each shard-file
- Do not access shard-files
if Bloom Filter returns
false for key

Chrome
Identifying
Malicious URLs
Store malicious
URLs in Bloom
Filter
In-depth check on
positive response

How it works
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

Basics of
Bloom
Filter
● Array of m bits
● initially set to 0
● Hash functions
● k hash functions
defined
● maps or hashes some
set element to one of
the m array positions

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
{ }m = 20
k = 3

0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0
{ cat}m = 20
k = 3

0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 0 1
{ cat , dog }m = 20
k = 3

0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1
{cat , dog , mouse }m = 20
k = 3

0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1
{cat , dog , mouse }
owl
m = 20
k = 3
owl is definitely NOT in set

0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1
rat
m = 20
k = 3
rat might be in set
{cat , dog , mouse }

1
2
3
4
5
PROBABILITIES
101
Probability of slot not selected for a single hash function
Probability of slot not selected for all k hashes
Probability of slot not selected for all k hashes of n elements
Probability of slot selected for all k hashes of n elements
Probability of k slots selected for n elements === False positive

If you want to read
more on the maths,
take a look at
https://en.wikipedia.org/
wiki/Bloom_filter

A Simple
Calculator
Click here: https://hur.st/bloomfilter

Next step
A Bloom what?
Why would I want one?
Practical examples
How it works
Next step

Bloom
Filters in
Code
● Libraries exist in most
languages
○ https://github.com/alexan
drnikitin/bloom-filter-scala
○ https://www.npmjs.com/p
ackage/bloom-filters
● Distributed Bloom Filters
in Redis
○ Perfect for multiple
instance of a service
using single bloom filter

Now, go think how using
Bloom Filters in your
services could reduce
payload and speed things
up

Other
probabilistic
structures
● Counting Bloom Filters
○ Allow deletes
● TopK
○ Keep track of Top
K counts of
elements
● Count-Min Sketch
○ Frequency table of
elements

The document discusses various data structures including skip lists, AVL trees, and hashing. It explains that skip lists allow for logarithmic-time operations and are simple to implement. Hashing provides constant-time operations by mapping keys to array indices via a hash function, but collisions must be handled. Common hash functions discussed include summing character codes or converting to a number in a prime base.

Java script array methodschauhankapil

Begin with Machine LearningNarong Intiruk

Tutorial 9 (bloom filters)Kira

Bloom filters are a space-efficient probabilistic data structure for representing a set in order to support membership queries. They allow for false positives but not false negatives. The document discusses how bloom filters work using hash functions to set bits in a bit vector, allowing for fast set membership checks. It also covers extensions like counting bloom filters that can support deletions by incrementing and decrementing counters, and variations like distance-sensitive bloom filters and bloomier filters.

Lecture_3.pptxGayathriSanthosh11

Bloom filters are a space-efficient probabilistic data structure for representing a set in order to support membership queries. A Bloom filter represents a set as a bit array and uses hash functions to map elements to bit positions, setting the bits. To query if an element is in the set, its bit positions are checked. False positives are possible but false negatives are not. Bloom filters enable representing a set using less space than a regular hash table while still supporting fast membership queries. They find many applications including caching, network routing, and database optimizations.

Approximate "Now" is Better Than Accurate "Later"NUS-ISS

How does Twitter track the top trending topics? How does Amazon keep track of the top-selling items for the day? How many cabs have been booked this month using your App? Is the password that a new user is choosing a common/compromised password? Modern web-scale systems process billions of transactions and generate terabytes of data every single day. In order to find answers to questions against this data, one would initiate a multi-minute query against a NoSQL datastore or kick off a batch job written in a distributed processing framework such as Spark or Flink. However, these jobs are throughput-heavy and not suited for realtime low-latency queries. However, you and your customers would like to have all this information "right now". At the end of this talk, you'll realize that you can power these low-latency queries and with incredibly low memory footprint "IF" you are willing to accept answers that are, say, 96-99% accurate. This talk introduces some of the go-to probabilistic data structures that are used by organisations with large amounts of data - specifically Bloom filter, Count Min Sketch and HyperLogLog.

New zealand bloom filterxlight

Bloom filters provide a space-efficient probabilistic data structure for representing a set in order to support membership queries. They allow false positives but no false negatives. The structure uses k hash functions to map elements to bit positions in a bit array. Querying whether an element is in the set checks if the corresponding bit positions are all set to 1. Modern applications include distributed caching, peer-to-peer networks, routing, and measurement infrastructure where Bloom filters trade off exact representation for speed and space efficiency.

Unit 5 Streams2.pptxSonaliAjankar

The document discusses streaming algorithms and Bloom filters. It defines the stream model as sequentially entering data at a rapid rate from multiple inputs where the entire stream cannot be stored, processing must be done in real-time with limited memory. Bloom filters are a space-efficient probabilistic data structure used to test set membership. Elements are hashed into a bit array, setting bits to 1. Querying an element checks if its hashed bits are set, returning true or a potential false positive if another element has set the bits. The probability of false positives decreases as the size of the bit array increases.

It Probably Works - QCon 2015Fastly

The document discusses probabilistic algorithms and their use for solving count-distinct problems. It explains that probabilistic algorithms provide probabilistically bounded error rates rather than exact answers. The count-distinct problem aims to estimate the number of unique elements in a large dataset and is used in applications like estimating unique website visitors. The document presents the LogLog algorithm as a probabilistic solution that partitions data and tracks the maximum number of leading zeros seen to estimate the count in a memory-efficient way.

#GDC15 Code ClinicMike Acton

VISUG - Approaches for application request throttlingMaarten Balliauw

Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ... In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.

Primer to Machine LearningJeff Tanner

This talk is a primer to Machine Learning. I will provide a brief introduction what is ML and how it works. I will walk you down the Machine Learning pipeline from data gathering, data normalizing and feature engineering, common supervised and unsupervised algorithms, training models, and delivering results to production. I will also provide recommendations to tools that help you provide the best ML experience, include programming languages and libraries. If there is time at the end of the talk, I will walk through two coding examples, using the HMS Titanic Passenger List, present with Python scikit-learn using algorithm random-trees to check if ML can correctly predict passenger survival and with R programming for feature engineering of the same dataset Note to data-scientists and programmers: If you sign up to attend, plan to visit my Github repository! I have many Machine Learning coding examples in Python scikit-learn, GNU Octave, and R Programming. https://github.com/jefftune/gitw-2017-ml

Tokens, Complex Systems, and NatureTrent McConaghy

Data oriented design and c++Mike Acton

The document discusses data-oriented design principles for game engine development in C++. It emphasizes understanding how data is represented and used to solve problems, rather than focusing on writing code. It provides examples of how restructuring code to better utilize data locality and cache lines can significantly improve performance by reducing cache misses. Booleans packed into structures are identified as having extremely low information density, wasting cache space.

Machine Learning in Python - PyLadies StockholmDhiana Deva

This document provides an overview of machine learning concepts and how Python can be used for machine learning. It discusses misconceptions about machine learning, the concepts of supervised and unsupervised learning, common machine learning algorithms like k-nearest neighbors and neural networks. It also shows how Python has useful tools for tasks like classification, dimensionality reduction, and data visualization that make it a good choice for machine learning applications.

Gc Atomicbufanliu

Garbage collection in Flash is triggered by memory allocation rather than deletion. It works by marking referenced objects as "in use" and clearing any unmarked objects. Since it is not predictable, memory usage may not return to its initial level after garbage collection. To test for memory leaks, automating repeatable sequences and checking for increasing memory usage over time is recommended. Event listeners can cause references in the opposite direction and need to be removed to prevent leaks.

Lunch session: Quantum ComputingRolf Huisman

The document discusses quantum computing and its potential uses and limitations. It begins with an explanation of how far a person could count in their lifetime compared to what conventional computing and quantum computing are capable of. It then covers the basics of quantum computing including qubits, quantum gates, quantum circuits, and measurement. Examples of different approaches to building quantum computers are provided. While progress is being made, quantum computing is still in its early stages with devices currently having just tens of qubits. The document concludes with a discussion of the investments and doubling of quantum volume needed each year to achieve quantum advantage in the 2020s and potential early uses in areas like cryptography.

Algorithms 101 for Data ScientistsChristopher Conlan

This document provides an overview of algorithm complexity and big O notation. It defines computational complexity as describing the amount of resources required for an algorithm relative to its inputs. Big O notation is introduced as a way to describe the growth rate of an algorithm's runtime. Common time complexities for algorithms like sorting, searching, and matrix operations are listed. The document warns of hidden complexity and provides examples of how easy it is to unintentionally write inefficient code. Benchmark results are shown for optimal versus naive implementations of algorithms like cumulative sums and word counting.

Lecture 2 - Bit vs Qubits.pptxNatKell

This document provides an overview of how classical and quantum computers work. It discusses how classical computers use binary digits (bits) to represent information, while quantum computers use quantum bits (qubits) that can exist in superpositions of states. The document outlines several problems like integer factorization and search that quantum computers may be able to solve faster using algorithms like Shor's and Grover's algorithms. It also notes some limitations of quantum computers in that they manipulate probabilities rather than definite states and only produce a single outcome when measured.

Practical Deep Learning Using Tensor Flow - Sandeep KathSandeep Kath

streamingalgo88585858585858585pppppp.pptxGopiNathVelivela

The document summarizes the Count-Min Sketch streaming algorithm. It uses a two-dimensional array and d independent hash functions to estimate item frequencies in a data stream using sublinear space. It works by incrementing the appropriate counters in each row when an item arrives. The estimated frequency of an item is the minimum value across the rows. Analysis shows that for an array width w proportional to 1/ε, the estimate will be within an additive error of ε times the total frequency with high probability.

A Gentle Introduction to Coding ... with PythonTariq Rashid

Evolving Search Relevancy: Presented by James Strassburg, Direct SupplyLucidworks

Streaming AlgorithmsJoe Kelley

Performance is a Feature!PostSharp Technologies

Starting with the premise that "Performance is a Feature", Matt Warren will show you how to measure, what to measure and how to get the best performance from your .NET code. We will look at real-world examples from the Roslyn code-base and StackOverflow (the product), including how the .NET Garbage Collector needs to be tamed! The presentation covers: Why we should care about performance Pitfalls to avoid when measuring performance How the .NET Garbage Collector can hurt performance Real-world performance lessons from open-source code The webinar recording can be found here: http://www.postsharp.net/blog/post/webinar-recording-performance-is-a-feature

Beyond lists - Copenhagen 2015Phillip Trelford

The document contains various notes and information snippets on topics including: - WiFi login details - Maslow's hierarchy of needs - LAMP stack vs 3-tier architecture - Performance impacts of small delays in page load times - Quicksort algorithm implementation in Haskell - Comparisons of Reactive Extensions, F# Observable, and Nessos Streams libraries - HybridDictionary data structure optimization in .NET - B+ tree data structure praise - Phillip Trelford's Twitter handle and blog

2018 Global Azure Bootcamp Azure Machine Learning for neural networksSetu Chokshi

This was the introduction session done for the 2018 Global Azure Bootcamp to get the users started with neural networks on Azure Machine Learning Studio. This gives them the initial introduction on how to develop and write the neural networks. We started with writing LeNet architecture on Azure Machine Learning studio to identify handwritten digits and then moved on to cats and dogs. This was also the presented in the first workshop of my meetup Microsoft Ai, ML Community which can be reached here https://www.meetup.com/Microsoft-AI-ML-Community/

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

tecnologias de las primeras civilizaciones.pdffjgm517

More Related Content

Similar to Introduction to Bloom Filters (20)

Unit 5 Streams2.pptxSonaliAjankar

It Probably Works - QCon 2015Fastly

#GDC15 Code ClinicMike Acton

VISUG - Approaches for application request throttlingMaarten Balliauw

Primer to Machine LearningJeff Tanner

Tokens, Complex Systems, and NatureTrent McConaghy

Data oriented design and c++Mike Acton

Machine Learning in Python - PyLadies StockholmDhiana Deva

Gc Atomicbufanliu

Lunch session: Quantum ComputingRolf Huisman

Algorithms 101 for Data ScientistsChristopher Conlan

Lecture 2 - Bit vs Qubits.pptxNatKell

Practical Deep Learning Using Tensor Flow - Sandeep KathSandeep Kath

streamingalgo88585858585858585pppppp.pptxGopiNathVelivela

A Gentle Introduction to Coding ... with PythonTariq Rashid

Evolving Search Relevancy: Presented by James Strassburg, Direct SupplyLucidworks

Streaming AlgorithmsJoe Kelley

Performance is a Feature!PostSharp Technologies

Beyond lists - Copenhagen 2015Phillip Trelford

2018 Global Azure Bootcamp Azure Machine Learning for neural networksSetu Chokshi

Unit 5 Streams2.pptxSonaliAjankar

It Probably Works - QCon 2015Fastly

#GDC15 Code ClinicMike Acton

VISUG - Approaches for application request throttlingMaarten Balliauw

Primer to Machine LearningJeff Tanner

Tokens, Complex Systems, and NatureTrent McConaghy

Data oriented design and c++Mike Acton

Machine Learning in Python - PyLadies StockholmDhiana Deva

Gc Atomicbufanliu

Lunch session: Quantum ComputingRolf Huisman

Algorithms 101 for Data ScientistsChristopher Conlan

Lecture 2 - Bit vs Qubits.pptxNatKell

Practical Deep Learning Using Tensor Flow - Sandeep KathSandeep Kath

streamingalgo88585858585858585pppppp.pptxGopiNathVelivela

A Gentle Introduction to Coding ... with PythonTariq Rashid

Evolving Search Relevancy: Presented by James Strassburg, Direct SupplyLucidworks

Streaming AlgorithmsJoe Kelley

Performance is a Feature!PostSharp Technologies

Beyond lists - Copenhagen 2015Phillip Trelford

2018 Global Azure Bootcamp Azure Machine Learning for neural networksSetu Chokshi

Recently uploaded (20)

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

tecnologias de las primeras civilizaciones.pdffjgm517

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

How Can I use the AI Hype in my Business Context?Daniel Lehner

𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨? Everyone’s talking about AI but is anyone really using it to create real value? Most companies want to leverage AI. Few know 𝗵𝗼𝘄. ✅ What exactly should you ask to find real AI opportunities? ✅ Which AI techniques actually fit your business? ✅ Is your data even ready for AI? If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.

Build Your Own Copilot & Agents For DevsBrian McKeiver

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

tecnologias de las primeras civilizaciones.pdffjgm517

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

ThousandEyes Partner Innovation Updates for May 2025ThousandEyes

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

How Can I use the AI Hype in my Business Context?Daniel Lehner

Build Your Own Copilot & Agents For DevsBrian McKeiver

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

Introduction to Bloom Filters

1. Hayden Marchant, Software Engineer [email protected] Bloom Filters

2. Agenda A Bloom what? Why would I want one? Practical examples How it works Next step

3. A Bloom-what? A Bloom what? Why would I want one? Practical examples How it works Next step

4. Just a Data Structure ‘Is an element in a set or not’

5. Probabilistic

6. Deﬁnitely not in set Probably in set OR

7. ILLUSTRATION

8. Why would I want one? A Bloom what? Why would I want one? Practical examples How it works Next step

9. Why not just store all used keys?

10. Element Key size: # elements: Memory required: 100 bytes 10 million 100 * 10M = 1GB SIMPLE MATHS

11. Bloom Filters: Tiny footprint ~ 10 bits per element

12. >98% reduction in space Element Key size: # elements: Memory required: 10 bits 10 million 100M bits ~= 12MB SIMPLE MATHS (PART 2)

13. Footprints: Traditional = 1GB Bloom Filter <=12MB

14. Practical Examples A Bloom what? Why would I want one? Practical examples How it works Next step

15. Akamai Prevent caching of One-Hit-Wonders 75% of web requests are only ever called once - Called One-Hit-Wonders Use Bloom Filters to prevent one-hit-wonders to be stored in Disk Cache 1. Saves disk-access for these 75% of requests 2. reducing workload

16. Medium Avoid recommending articles a user has read - Add article recommended to bloom filter - $userid-$articleid - Before recommending, check in Bloom Filter if (user,article) exists. - If not, then recommend and then add to bloom filter

17. Cassandra Reduce disk lookup for non-existent rows in file storage - Shard responsible over range of data - Each shard stored in immutable shard-files - Create Bloom Filter in server for each shard-file - Do not access shard-files if Bloom Filter returns false for key

18. Chrome Identifying Malicious URLs Store malicious URLs in Bloom Filter In-depth check on positive response

19. How it works A Bloom what? Why would I want one? Practical examples How it works Next step

20. Basics of Bloom Filter ● Array of m bits ● initially set to 0 ● Hash functions ● k hash functions defined ● maps or hashes some set element to one of the m array positions

21. Bloom Filter in slow motion

22. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 { }m = 20 k = 3

23. 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 { cat}m = 20 k = 3

24. 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 0 1 { cat , dog }m = 20 k = 3

25. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1 {cat , dog , mouse }m = 20 k = 3

26. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1 {cat , dog , mouse } owl m = 20 k = 3 owl is definitely NOT in set

27. 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 1 1 0 1 rat m = 20 k = 3 rat might be in set {cat , dog , mouse }

28. Constant memory space

29. Extremely Fast Processing

30. Low Error Rate

31. The maths ‘Under-the-hood’

32. 1 2 3 4 5 PROBABILITIES 101 Probability of slot not selected for a single hash function Probability of slot not selected for all k hashes Probability of slot not selected for all k hashes of n elements Probability of slot selected for all k hashes of n elements Probability of k slots selected for n elements === False positive

33. GETTING A LITTLE TRICKIER

34. If you want to read more on the maths, take a look at https://en.wikipedia.org/ wiki/Bloom_filter

35. A Simple Calculator Click here: https://hur.st/bloomfilter

36. Next step A Bloom what? Why would I want one? Practical examples How it works Next step

37. Bloom Filters in Code ● Libraries exist in most languages ○ https://github.com/alexan drnikitin/bloom-filter-scala ○ https://www.npmjs.com/p ackage/bloom-filters ● Distributed Bloom Filters in Redis ○ Perfect for multiple instance of a service using single bloom filter

38. Now, go think how using Bloom Filters in your services could reduce payload and speed things up

39. Other probabilistic structures ● Counting Bloom Filters ○ Allow deletes ● TopK ○ Keep track of Top K counts of elements ● Count-Min Sketch ○ Frequency table of elements

40. Q&A

Introduction to Bloom Filters

Recommended

More Related Content

Similar to Introduction to Bloom Filters (20)

Recently uploaded (20)

Introduction to Bloom Filters