BigData in BlockChains
Download as PPTX, PDF1 like170 views
Blockchains provide a fertile ground for storing and managing big data in a decentralized manner. Some examples of using blockchains for big data include Numer.ai for storing machine learning data shapes, IPFS and FileCoin for distributed storage, and BigchainDB as a distributed database with blockchain characteristics. Blockchains allow users to store and control their own data while granting view rights to interested and trusted parties, making it even GDPR compliant by keeping users as the data controllers.
Ad
More Related Content
Similar to BigData in BlockChains (20)
More from Felix Crisan (15)
Ad
Recently uploaded (20)
Ad
BigData in BlockChains
- 1. (Big) Data in (Block) Chains Felix Crisan
- 2. About Me NETOPIA / mobilPay Big Data Blockchain Blockchain Romania Community @fixone on almost every platform
- 3. Trivia
- 4. Problem statement Storage Computing Monolithic “Regular” disk (diskette, USB drive) “Regular” processor Distributed Disks in RAID, NAS, SAN MapReduce (?!)
- 5. Distributed database = storage + more than one processor (or, these days, computer)
- 6. Distributed database = homogenous* nodes
- 7. CAP Theorem Postulated by Eric Brewer in 1999 Mathematical proof by Gilbert and Lynch in 2002
- 8. Consistency, Availability, Partition Tolerance Any read must reflect any (related) prior write Any live node must respond (correctly!) in a finite (predefined?) amount of time The system continues to work even if network is partitioned
- 9. Consistency, Availability, Partition Tolerance -> pick any two
- 11. Distributed ledger = heterogeneous distributed database (usually mutable!)
- 12. Blockchain = guaranteed (cryptography FTW!) immutable DL with no trust assumptions
- 13. Evolution of data structures ● File -> Database ● Database -> Distributed Database ● Distributed Database -> Distributed Ledger ● Distributed Ledger -> Blockchain
- 14. Did anyone say “blockchain”?
- 15. Fallacies of distributed computing (by Peter Deutsch) The network is reliable. There is one administrator. Infinite bandwidth. The network is secure. Topology doesn’t change. Latency is zero. Transport cost is zero. The network is homogeneous.
- 16. Latency - network & computation Asynchrony: no upper bound Synchrony: there’s an upper bound Partial-synchrony: there’s an upper bound but it is not known beforehand
- 17. Byzantine failure Failure models Omission failure (drop messages) Crash failure (stop functioning)
- 18. Entity (e.g. node) actions Respond correctly (0=>0, 1=>1) Does not respond (0=>Φ, 1=>Φ) Respond falsely (0=>1, 1=>0)
- 19. Distributed systems Adversarial (Byzantine) vs non-adversarial “value replication” vs “state machine replication” (e.g. RBR vs SBR in MySQL) Don’t trust, verify (back in the days they used to say “Nullius in verba”)
- 20. The “holy trinity” Safety: only good things happen (i.e. bad things don’t happen) Liveness: a good thing will eventually happen Fault-tolerance: can function with components down
- 21. Consensus components (1/2) Integrity: all correct processes decide at most one value v, and v is the “right” value (safety property). Agreement: all correct processes must agree on the same value (safety property).
- 22. Consensus components (2/2) Termination: all correct processes decide some value (liveness property). Validity: if all correct processes decide v, then v must have been proposed by some correct process (non-triviality property).
- 23. FLP impossibility: fault-tolerant agreement is impossible in an asynchronous system
- 24. Otherwise put: distributed consensus is impossible if one process fails
- 25. This means that all distributed systems have trade-offs
- 27. Fun Facts FLP just stands for Fischer, Lynch (the same from Gilbert/Lynch!), Paterson The term “Byzantine” (with the meaning “adversarial”) was introduced in 1978 in a work by Leslie Lamport and Robert Shostak Lamport was roommate with Whitfield Diffie (Diffie-Hellman anyone?) in the ‘70s
- 28. Conflict resolution In distributed databases: through a nonce or the timestamp of the operation In blockchains - consensus protocol: ○ PoW: through probabilistic consensus (Nakamoto consensus) - “most work” wins ○ PoS: crypto-economic incentives and disincentives
- 29. CAP vs FLP? CAP theorem represents a trade-off between liveness and safety Availability ~ Liveness Consistency ~ Safety CAP is a particular case of FLP
- 30. Let’s complicate things even more Public vs Private blockchain (e.g. Bitcoin vs Hyperledger) Permissionless vs Permissioned (e.g Ethereum vs Corda)
- 32. Okay, so when do I need a blockchain?
- 33. Almost never.
- 34. That is if you don’t need a database that Is decentralized Is immutable Is tamper proof Works with untrusted participants (i.e. suitable for value transfer networks)
- 36. Data in Chains
- 38. Evolution of value Up to 19th century-> land Afterwards, until now-> capital Future-> data (some say)
- 39. Not only data, also metadata ( = governance + consensus )
- 43. Blockchains provide a fertile ground for data applications and data storage
- 44. (however, beware of the pseudonimity in public blockchains)
- 45. For instance Numer.ai (Fan and Vercauteren homomorphic encryption scheme, because ML only cares about data “shape”) IPFS: distributed storage FileCoin: distributed storage with Proof of Replication and Proof of Storage
- 46. For instance BigchainDB (big data distributed database with blockchain characteristics) OrbitDB (serverless, distributed, peer- to-peer database)
- 47. For instance The Graph (GraphQL for Web3) Ocean Protocol (data economy, secure expose data) SingularityNET (Decentralized Artificial Intelligence Marketplace - “AI services and algorithms”)
- 48. Yes, blockchain allows users to store and control their own data
- 49. And they can give interested (trusted) parties view rights (using viewing keys)
- 50. … and it’s even GDPR compliant (users themselves are the data controllers)
- 51. It’s actually already working
- 52. Congrats, you made it so far!