Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by Daniel Lemire
3 likes1,834 views
Contemporary computing hardware offers massive new performance opportunities. Yet high-performance programming remains a daunting challenge. We present some of the lessons learned while designing faster indexes, with a particular emphasis on compressed bitmap indexes. Compressed bitmap indexes accelerate queries in popular systems such as Apache Spark, Git, Elastic, Druid and Apache Kylin.
![Add and contains on bitset
Most of the processors work on 64‑bit words.
Given index x , the corresponding word index is x/64 and within‑
word bit index is x % 64 .
add(x) {
array[x / 64] |= (1 << (x % 64))
}
contains(x) {
return array[x / 64] & (1 << (x % 64))
}
8](https://image.slidesharecdn.com/7bdaniellemire-170215222842/85/Engineering-Fast-Indexes-for-Big-Data-Applications-Spark-Summit-East-talk-by-Daniel-Lemire-8-320.jpg)
![How fast can you set bits in a bitset?
Very fast! Roughly three instructions (on x64)...
index = x / 64 -> a single shift
mask = 1 << ( x % 64) -> a single shift
array[ index ] |- mask -> a logical OR to memory
(Or can use BMI's bts .)
On recent x64 can set one bit every ≈ 1.65 cycles (in cache)
Recall : Modern processors are superscalar (more than one
instruction per cycle)
9](https://image.slidesharecdn.com/7bdaniellemire-170215222842/85/Engineering-Fast-Indexes-for-Big-Data-Applications-Spark-Summit-East-talk-by-Daniel-Lemire-9-320.jpg)
![Bitsets are efficient: in practice
for i in [0...n]
out[i] = A[i] & B[i]
Recent x64 processors can do this at a speed of ≈ 0.5 cycles per
pair of input 64‑bit words (in cache) for n = 1024 .
0.5
memcpy runs at ≈ 0.3 cycles.
0.3
11](https://image.slidesharecdn.com/7bdaniellemire-170215222842/85/Engineering-Fast-Indexes-for-Big-Data-Applications-Spark-Summit-East-talk-by-Daniel-Lemire-11-320.jpg)
![Hybrid Model
Decompose 32‑bit space into
16‑bit spaces (chunk).
Given value x, its chunk index is x ÷ 2 (16 most significant bits).
For each chunk, use best container to store least 16 significant bits:
a sorted array ({1,20,144})
a bitset (0b10000101011)
a sequences of sorted runs ([0,10],[15,20])
That's Roaring!
Prior work: O'Neil's RIDBit + BitMagic
16
22](https://image.slidesharecdn.com/7bdaniellemire-170215222842/85/Engineering-Fast-Indexes-for-Big-Data-Applications-Spark-Summit-East-talk-by-Daniel-Lemire-22-320.jpg)
![[164] pinpoint](https://cdn.slidesharecdn.com/ss_thumbnails/164pinpoint-150914054628-lva1-app6892-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by Daniel Lemire (20)
Ad
More from Spark Summit (20)
Recently uploaded (20)
Engineering Fast Indexes for Big-Data Applications: Spark Summit East talk by Daniel Lemire
- 1. ENGINEERING FAST INDEXES Daniel Lemire https://lemire.me Joint work with lots of super smart people
- 2. Our recent work: Roaring Bitmaps http://roaringbitmap.org/ Used by Apache Spark, Netflix Atlas, LinkedIn Pinot, Apache Lucene, Whoosh, Metamarket's Druid eBay's Apache Kylin Further reading: Frame of Reference and Roaring Bitmaps (at Elastic, the company behind Elasticsearch) 2
- 3. Set data structures We focus on sets of integers: S = {1, 2, 3, 1000}. Ubiquitous in database or search engines. tests: x ∈ S? intersections: S ∩ S unions: S ∪ S differences: S ∖ S Jaccard Index (Tanimoto similarity) ∣S ∩ S ∣/∣S ∪ S ∣ 2 1 2 1 2 1 1 1 1 2 3
- 4. "Ordered" Set iterate in sorted order, in reverse order, skippable iterators (jump to first value ≥ x) Rank: how many elements of the set are smaller than k? Select: find the kth smallest value Min/max: find the maximal and minimal value 4
- 5. Let us make some assumptions... Many sets containing more than a few integers Integers span a wide range (e.g., [0, 100000)) Mostly immutable (read often, write rarely) 5
- 6. How do we implement integer sets? Assume sets are mostly imutable. sorted arrays ( std::vector<uint32_t> ) hash sets ( java.util.HashSet<Integer> , std::unordered_set<uint32_t> ) … bitsets ( java.util.BitSet ) compressed bitsets 6
- 7. What is a bitset??? Efficient way to represent a set of integers. E.g., 0, 1, 3, 4 becomes 0b11011 or "27". Also called a "bitmap" or a "bit array". 7
- 8. Add and contains on bitset Most of the processors work on 64‑bit words. Given index x , the corresponding word index is x/64 and within‑ word bit index is x % 64 . add(x) { array[x / 64] |= (1 << (x % 64)) } contains(x) { return array[x / 64] & (1 << (x % 64)) } 8
- 9. How fast can you set bits in a bitset? Very fast! Roughly three instructions (on x64)... index = x / 64 -> a single shift mask = 1 << ( x % 64) -> a single shift array[ index ] |- mask -> a logical OR to memory (Or can use BMI's bts .) On recent x64 can set one bit every ≈ 1.65 cycles (in cache) Recall : Modern processors are superscalar (more than one instruction per cycle) 9
- 10. Bit‑level parallelism Bitsets are efficient: intersections Intersection between {0, 1, 3} and {1, 3} can be computed as AND operation between 0b1011 and 0b1010 . Result is 0b1010 or {1, 3}. Enables Branchless processing. 10
- 11. Bitsets are efficient: in practice for i in [0...n] out[i] = A[i] & B[i] Recent x64 processors can do this at a speed of ≈ 0.5 cycles per pair of input 64‑bit words (in cache) for n = 1024 . 0.5 memcpy runs at ≈ 0.3 cycles. 0.3 11
- 12. Bitsets can be inefficient Relatively wasteful to represent {1, 32000, 64000} with a bitset. Would use 1000 bytes to store 3 numbers. So we use compression... 12
- 13. Memory usage example dataset : census1881_srt format bits per value hash sets 200 arrays 32 bitsets 900 compressed bitsets (Roaring) 2 https://github.com/RoaringBitmap/CBitmapCompetition 13
- 14. Performance example (unions) dataset : census1881_srt format CPU cycles per value hash sets 200 arrays 6 bitsets 30 compressed bitsets (Roaring) 1 https://github.com/RoaringBitmap/CBitmapCompetition 14
- 15. What is happening? (Bitsets) Bitsets are often best... except if data is very sparse (lots of 0s). Then you spend a lot of time scanning zeros. Large memory usage Bad performance Threshold? ~1 100 15
- 16. Hash sets are not always fast Hash sets have great one‑value look‑up. But they have poor data locality and non‑trivial overhead... h1 <- some hash set h2 <- some hash set ... for(x in h1) { insert x in h2 // "sure" to hit a new cache line!!!! } 16
- 17. Want to kill Swift? Swift is Apple's new language. Try this: var d = Set<Int>() for i in 1...size { d.insert(i) } // var z = Set<Int>() for i in d { z.insert(i) } This blows up! Quadratic‑time. Same problem with Rust. 17
- 18. What is happening? (Arrays) Arrays are your friends. Reliable. Simple. Economical. But... binary search is branchy and has bad locality... while (low <= high) { int middleIndex = (low + high) >>> 1; int middleValue = array.get(middleIndex); if (middleValue < ikey) { low = middleIndex + 1; } else if (middleValue > ikey) { high = middleIndex - 1; } else { return middleIndex; } } return -(low + 1); 18
- 19. Performance: value lookups (x ∈ S) dataset : weather_sept_85 format CPU cycles per query hash sets ( std::unordered_set ) 50 arrays 900 bitsets 4 compressed bitsets (Roaring) 80 19
- 20. How do you compress bitsets? We have long runs of 0s or 1s. Use run‑length encoding (RLE) Example: 000000001111111100 can be coded as 00000000 − 11111111 − 00 or <5><1> using the format < number of repetitions >< value being repeated > 20
- 21. RLE‑compressed bitsets Oracle's BBC WAH (FastBit) EWAH (Git + Apache Hive) Concise (Druid) … Further reading: http://githubengineering.com/counting‑objects/ 21
- 22. Hybrid Model Decompose 32‑bit space into 16‑bit spaces (chunk). Given value x, its chunk index is x ÷ 2 (16 most significant bits). For each chunk, use best container to store least 16 significant bits: a sorted array ({1,20,144}) a bitset (0b10000101011) a sequences of sorted runs ([0,10],[15,20]) That's Roaring! Prior work: O'Neil's RIDBit + BitMagic 16 22
- 23. Roaring All containers fit in 8 kB (several fit in L1 cache) Attempts to select the best container as you build the bitmaps Calling runOptimize will scan (quickly!) non‑run containers and try to convert them to run containers 23
- 24. Performance: union (weather_sept_85) format CPU cycles per value bitsets 0.6 WAH 4 EWAH 2 Concise 5 Roaring 0.6 24
- 25. What helps us... All modern processors have fast population‑count functions ( popcnt ) to count the number of 1s in a word. Cheap to keep track of the number of values stored in a bitset! Choice between array, run and bitset covers many use cases! 25
- 26. Go try it out! Java, Go, C, C++, C#, Rust, Python... (soon: Swift) http://roaringbitmap.org Documented interoperable serialized format. Free. Well‑tested. Benchmarked. Peer reviewed Consistently faster and smaller compressed bitmaps with Roaring. Softw., Pract. Exper. (2016) Better bitmap performance with Roaring bitmaps. Softw., Pract. Exper. (2016) Optimizing Druid with Roaring bitmaps, IDEAS 2016, 2016 Wide community (dozens of contributors). 26