New Neo4j Auto HA Cluster
7 likesâ˘4,755 views
The document discusses the Neo4j high availability (HA) auto-cluster system, which replaces the previous use of Apache Zookeeper by implementing a multi-Paxos consensus algorithm for cluster management and master election. This approach simplifies the architecture by addressing previous challenges such as unreliable operations and the need for separate components. The implementation focuses on using state machines for HA functionality, including notifications, heartbeat management, and reliability through timeouts.
New Neo4j Auto HA Cluster
- 1. Neo4j High Availability New Auto-Cluster Michael Hunger - @mesirii 1
- 2. High Availability Cluster ŕšNeo4j Enterprise ŕšMaster-Slave Replication ŕšread-scaling and fault-tolerance ŕševentual consistency ⢠write to master (push_factor) ⢠write to slaves 2
- 3. 3 Separate Concerns (I) ŕšCluster Management ⢠Members join/leave/heartbeat ŕšFailover ⢠Master Election ⢠Distribution of Master-Status 3
- 4. 3 Separate Concerns (II) ŕšReplication â˘synchronized id-generation ⢠distributed locks ⢠pull, push of transactions ⢠initial store synchronization 4
- 5. Pre 1.9 - Zookeeper 5
- 6. Pre 1.9 ŕšApache Zookeeper took care of concerns ⢠Cluster Management âŁnew members register with ZK ⢠Failover âŁZK stores Master and last TX-Id âŁZK uses ZAB to determine new Master and distribute information 6
- 7. HA Cluster Coordinator RO- Coordinator Slave Master Slave Slave Coordinator 7
- 8. Pre 1.9 - Problems ŕšAdditional setup and operations of a separate component ŕšunreliable operation / hiccups ŕšlongterm stability ŕšno dynamic reconďŹg of the ZK cluster important for cloud setup 8
- 9. Post 1.9 - Neo4j Auto Cluster 9
- 10. Replace Zookeeper!? ŕšImplement Multi-Paxos ourselves ŕšsimple, testable code ŕšonly covers ⢠cluster management, ⢠master election 10
- 11. HA Cluster 11
- 12. What is Paxos? ŕšreliable consensus making ŕšbroadcasting ŕšworks even with unreliable communication â˘message lost ⢠delays, invalid order ŕšdoes not guarantee progress 12
- 13. What is Paxos? 13
- 14. Implementation ŕševerything is a State Machines ⢠SM = stateless enums + context ⢠Message = type enum + payload ⢠State = enum instance ⢠switch on msg-type, implement logic Transition = handle() messages, 14
- 15. Implementation (II) ŕševerything is a State Machines ⢠use timeouts for reliability ⢠handle failing messages ⢠decouple network and time âŁfor testability ⢠listeners interact on messages with outside world, sync or async 15
- 16. Implementation (II) ŕšPaxos (3 roles) Acceptor ⢠Proposer-SM Paxos ⢠Acceptor-SM Proposer Learner ⢠Learner-SM ClusterState ŕšCluster ⢠Heartbeat Heartbeat 16
- 17. Multi-Paxos (happy path) Acceptor Learner Proposer (2 * f + 1) PREPARE PREPARE TIMEOUT VALUE PROMISE MATCH OR REJECT NO MATCH ACCEPT MATCHES TIMEOUT PROMISE? CHECK , STORE STORE ACCEPTED VALUE RESPONSES OR IF QUORUM REJECTED NO MET, CANCEL TIMEOUT STORE ... VALUE LEARN OUT OF ORDER MSG HANDLING other DELIVER A VALUE IS Learner ALL VALID MISSING ATOMIC BC LEARN TIMEOUT WE STILL 17 LEARN TIMEOUT DON'T KNOW
- 18. TIMEOUT Multi-Paxos (happy path) PROMISE ACCEPT ... MATCHES TIMEOUT PROMISE? CHECK , STORE STORE ACCEPTED VALUE RESPONSES OR IF QUORUM REJECTED NO MET, CANCEL TIMEOUT STORE VALUE LEARN OUT OF ORDER MSG HANDLING other DELIVER A VALUE IS Learner ALL VALID MISSING ATOMIC BC LEARN TIMEOUT WE STILL LEARN TIMEOUT DON'T KNOW LEARN REQ LEARN TIMEOUT HAVE LEARN VALUE OR LEARN FAIL DON'T KNOW 18
- 19. Acceptor State Machine 19
- 20. Heartbeat State Machine 20
- 21. Implementation (III) ŕšHA Implementation uses state machines as infrastructure ŕšnotiďŹcations via listeners ŕšpiggyback heartbeat on messages ŕšmaster election ⢠(all - failed) have to agree ⢠Paxos BC needs quorum of total 21
- 22. Multi-Paxos ŕševerything is a State Machines ⢠use timeouts for reliability ⢠handle failing messages ⢠decouple network and time âŁfor testability ⢠listeners interact on messages with outside world, sync or async 22
- 23. Unit-Testing ⢠Mock Time âŁfast running tests despite timeouts ⢠Mock Network âŁsimulate delays, failing messages 23
- 24. Unit-Test-Example 24
- 25. Setup â˘ConďŹg ⢠Video ⢠Auto-Setup Script (Demo) 25
- 26. Thank You - Questions? 26