QuadIron An open source library for number theoretic transform-based erasure codes
1 like2,152 views
QuadIron is an open source library which implements erasure codes in a way that can scale better than any other existing system.
Ad
More Related Content
What's hot (20)
Similar to QuadIron An open source library for number theoretic transform-based erasure codes (20)
Ad
More from Scality (13)
Ad
Recently uploaded (20)
QuadIron An open source library for number theoretic transform-based erasure codes
- 1. A Library for Number Theoretic Transforms Erasure Codes Zenko Live: QuadIron
- 2. Agenda • Introduction to Zenko • The problem space of data resiliency • Giorgio Regni, CTO • QuadIron demo • Vianney Rancurel, R&D • Questions and Answers What’s Coming
- 3. QuadIron
- 4. Why? Planetary scale decentralized storage: • Distributing data over hundreds of drives, servers, locations with minimum overhead • Guaranteeing that each parities is useful and can help reconstruct the original data • Keeping the data secure even though fragments are present in many different places
- 5. Pro Tip: When you get stuck, change vectorial space To be updated by Giorgio
- 6. What happens when you need more parities?
- 7. Demo • Video file of ~90MB • Using coding 90+160 • Split in 90 fragments of 1MB: x.00 … x.89 • Generate 160 parities (in fact 250) (overhead of 2.77): x.c00 … x.c249 (non-systematic code) • Delete data fragments • Delete 100 parities (tolerate 100 drive failures!) • Repair • Play the video
- 8. QuadIron
- 9. Lib QuadIron is online Forum https://forum.zenko.io Website: www.zenko.io/blog Code: https://github.com/scality/quadiron https://www.zenko.io/blog/free-library-erasure-codes/
- 11. Backup
- 12. Properties of Erasure Codes: Definition A C(n,k) erasure code is defined by n=k+m ❒ k being the number of data fragments. ❒ m being the number of desired erasure fragments. Example: C(9, 6)
- 13. Properties of Erasure Codes ❒ Optimality: e.g. MDS (Maximum Distance Separable) erasure code guarantees that any k fragments can be used to decode a file ❒ Systematicity: Systematic codes generate n-k erasure fragments and therefore maintain k data fragments. Non-systematic codes generate n erasure fragments ❒ Speed: Erasure codes are characterized by their encode/decode speed. Speed may vary acc/to the rate (k and m parameters). Speeds may also be more or less predictive acc/to codes. ❒ Rate sensitivity: Erasure codes can also be compared by their sensitivity to the rate r=k/n, which may or may not impact the encoding and decoding speed ❒ Rate adaptivity: Changing k and m without having to generate all the erasure codes ❒ Confidentiality: determined if an attacker can partially decode the data if he obtains less than k fragments. Non-systematic codes are confidential (different from threshold schemes) ❒ Repair Bandwidth: the number of fragments required to repair a fragment.
- 14. (Main) Types of Erasure Codes ❒ Traditional RS Codes (e.g. Vandermonde or Cauchy matrices) ❒ LDPC Codes ❒ Locally-Repairable-Codes (LRC) ❒ FFT Based RS Codes ❒ Multiplicative FFTs (prime fields) ❒ Additive FFTs (binary extension fields)
- 15. Types of Codes: Traditional RS Codes
- 16. Types of Codes: Traditional RS Codes
- 17. Types of Codes: Traditional RS Codes The good: ❒ Simple ❒ Support systematic and adaptive rates. The bad: ❒ Matrix multiplication: O(k x n)
- 18. Types of Codes: LDPC Codes ❏ H is a matrix for a C(8,4) code ❏ wc is the number of 1 in a col ❏ wr is the number of 1s in a row ❏ To be called low density wc << n and wr << m ❏ Regular if wc constant and wr = wc .(n/m) ❏ Matrix can be generated pseudo-randomly ❏ Presence of short cycles f1, f2 bad Source: Bernhard M.J. Leiner
- 19. Types of Codes: LDPC Codes Low-Density-Parity-Check (LDPC) codes are also an important class of erasure codes and are constructed over sparse parity-check matrices. The good: ❒ Theoretically an LDPC code optimal for all the interesting properties for a given use case exist. The bad: ❒ LDPC are not MDS: it is always possible to find a pattern that cannot decode (e.g. having only k fragments out of n). Overhead is k*f or k+f with a small f, but the overhead is not deterministic. ❒ You can always find/design an LDPC code optimized for few properties (i.e. tailored for a specific use case) but it will be sub-optimal for the other properties ❒ Designing a good LDPC code is some kind of black art that requires a lot of fine tuning and experimentation.
- 20. Types of Codes: LRC Codes ❏ P1, P2, P3 and P4 are constructed over a standard RS ❏ S1 + S2 + S3 = 0 ❏ No need to store S3 Source: XORing Elephants: Novel Erasure Codes for Big Data
- 21. Types of Codes: LRC Codes Locally-Repairable-Codes (LRC) have tackled the repair bandwidth issue of the RS codes. They combine multiple layers of RS: the local codes and the global codes. The good: ❒ Better repair bandwidth than RS codes. Because with RS code we need to read k fragments to decode. The bad: ❒ Those codes are not MDS and they require an higher storage overhead than MDS codes.
- 22. Types of Codes: Multiplicative FFT
- 23. Types of Codes: Multiplicative FFT
- 24. Types of Codes: Additive FFT
- 25. Types of Codes: FFT Based RS Codes Fast Fourier transform (FFT) have a good set of desirable properties. The good: ❒ Relatively simple ❒ O(N.log(N)) (because we use FFT to speed up the matrix multiplication) ❒ MDS ❒ Fast for large n The bad: ❒ Repair bandwidth: If there is a missing erasure, we need k codes to recover the data fragments. For systematic codes, in any case we need to download k codes.
- 27. Multiplicative FFT: Horizontal Vectorization
- 28. Multiplicative FFT: Vertical Vectorization
- 29. Multiplicative FFT: Vertical Vectorization
- 30. Multiplicative FFT: Vertical Vectorization
- 31. Multiplicative FFT: Vertical Vectorization
- 32. Speed Comparison ❏ Isa-l: Intel Intelligent Storage Acceleration Library. Matrix based RS HW accelerated: http://01.org/intel-storage-acceleration-library-open-source-version ❏ Wirehair: Fast and Portable Fountain Codes in C. Hybrid LDPC. https://github.com/catid/wirehair ❏ Leopard: MDS Reed-Solomon Erasure Correction Codes for Large Data in C. Additive FFT based. https://github.com/catid/leopard Thanks Catid !
- 33. Types of Codes: Speed Comparison
- 34. Types of Codes: Speed Comparison
- 35. Types of Codes: Speed Comparison
- 36. Types of Codes: Speed Comparison
- 37. Types of Codes: Speed Comparison
- 38. Types of Codes: Speed Comparison
- 39. Types of Codes: Speed Comparison
- 40. Types of Codes: Speed Comparison
- 42. Application: Decentralized Storage Requirements for an erasure code for a decentralized storage archive: ❒ Simple (e.g. may compile on WASM) ❒ Fast, e.g. for > 24 fragments ❒ MDS: A rock solid contract ❒ Work with all rates, and all combinations of n and k ❒ Systematic for smaller fragments ❒ Non-systematic for larger fragments -> Confidentiality ensured if fragments not stored on same servers (not a threshold scheme though, must be combined with encryption) ❒ Repair-Bandwidth not critical
- 44. Application: Decentralized Storage ❒ Multiple locations, multiple servers per location ❒ Each server is a “Quadiron Provider” ❒ E.g. 10 locations on the globe with 5 servers/location: C(50,35) => can lose 3 locations or 15 servers for an overhead of 1.4 ❒ A server is just a bunch of disks, e.g. 45 drives ❒ Can have local parities on servers to avoid repairing too often on the network e.g. C(45, 40) = 1.125 ❒ Total overhead 1.4 * 1.125 = 1.57 ❒ E.g. w/ 10TB drives, 22PB => 14PB useful ❒ Use blockchain transactions to store the location of blocks ❒ E.g. using Parity, proof-of-work (non-trusted env) or proof-of-authority (trusted env => millions tx/s) ❒ Index the ledgers by block-ids ❒ Use the indexes to locate the blocks ❒ Consolidate indexes
- 45. Decentralized Storage: Zenko QuadIron ❏ Multi-cloud data controller ❏ 1 API endpoint S3 compatible ❏ Native cloud storage ❏ Metadata search across clouds ❏ 100% open source ❏ github.com/scality/zenko ❏ zenko.io ❏ forum.zenko.io ❏ Give us feedback ! ❏ Try the sandbox on Orbit ! S3 API Wasabi, Digital Ocean, etc
- 47. Using the Library C++ Library is available at: https://github.com/scality/quadiron LICENSE: BSD 3-clause Compiling: $ mkdir build $ cd build $ cmake -G 'Unix Makefiles' .. $ make
- 48. Using the Library: Code // #include <quadiron.h> const int word_size = 8; const int n_data = 16; const int n_parities = 64; const size_t pkt_size = 1024; quadiron::fec::RsFnt<T>* fec = new quadiron::fec::RsFnt<uint64_t>( quadiron::fec::FecType::NON_SYSTEMATIC, word_size, n_data, n_parities, pkt_size ); // encode std::vector<std::istream*> d_files(fec->n_data, nullptr); std::vector<std::ostream*> c_files(fec->n_outputs, nullptr); std::vector<quadiron::Properties> c_props(fec->n_outputs); fec->encode_packet(d_files, c_files, c_props); // decode std::vector<std::ostream*> r_files(fec->n_data, nullptr); fec->decode_bufs(d_files, c_files, c_props, r_files);
- 49. Using the Library: Next Steps ❒ Optimize Multiplicative FFT decoding: For now a relatively slow Lagrange interpolation ❒ We know how to do it for special values of k and m (k mod m = 0, m mod k = 0) ❒ Optimize Additive FFTs ❒ Implement Systematic Additives FFTs ❒ Implement NTT adaptive codes for both multiplicative and additive FFTs ❒ Other optimizations ❒ Frobenius FFTs for both multiplicative and additive FFTs
- 50. Developers Lam Pham-Sy: Lam Pham-Sy is a research engineer working on information theory and computer science. His main research focuses on different families of forward erasure correcting codes such as ReedSolomon codes, Low-Density Parity-Check codes, Locally Repairable codes etc. Their application covers from digital communication to data storage. He did his PhD program in a collaboration between CEA-Leti and Eutelsat S.A. on the subject of forward erasure codes for satellite communications. Afterwards, he continued his researches at ETIS laboratory and at Orange Labs. Currently he works at Scality S.A. as a research engineer whose research topics include application of erasure codes in distributed storage systems, finite field arithmetics. Sylvain Laperche: Sylvain Laperche is a code craftsman. With a background in biotech engineering,he learnt how to hack bacteria before learning how to hack a computer. That changed when it studied bioinformatics, and since then he honed and applied its skill on a wide set of problematics: genome sequencing, complex embedded systems, climate modelling at European scale, mass-scale geolocation for telco industries. Its steps led him to work on distributed storage systems and he currently works as an R&D engineer at Scality. Sylvain Laperche has an Engineer’s degree in Hardware, Circuit Design and Embedded Systems from ISIMA.
- 51. Zenko/QuadIron Community - Questions ? 1,000+ registered Zenko Orbit users Forum https://forum.zenko.io Website: www.zenko.io/blog QuadIron github: https://github.com/scality/quadiron https://www.zenko.io/blog/free-library-erasure-codes/