A Pulsar Use Case In Federated Learning - Pulsar Summit NA 2021
Download as PPTX, PDF0 likes326 views
Federated learning (FL) is a machine learning technique that enables multiple decentralized organizations to train a model without exposing local data samples. Instead, during the training, lots of encrypted messages will be exchanged among the participants to aggregate the global model. Due to the message is so important and its requirements of real-time and sequential, it brings some challenges to the transmission. In this session, we will talk about how to address the above challenge with the Apache Pulsar project, and we will go through the details about how popular FL project FATE(https://github.com/FederatedAI/FATE) use Pulsar to do federated training.
Ad
More Related Content
What's hot (20)
Similar to A Pulsar Use Case In Federated Learning - Pulsar Summit NA 2021 (20)
Ad
More from StreamNative (20)
Ad
Recently uploaded (20)
A Pulsar Use Case In Federated Learning - Pulsar Summit NA 2021
- 1. ©2020 VMware, Inc. A Pulsar Use Case In Federated Learning Pulsar Summit 2021 Jiahao Chen MTS, VMware
- 2. 2 ©2020 VMware, Inc. About me Jiahao Chen, software engineer at VMware, specializes in container, network and distributed technology research, and is also actively involved in the construction of open-source communities. During his tenure at VMware, he led or participated in the completion of the construction of multiple platform-level projects
- 3. Agenda 3 ©2020 VMware, Inc. 1. Background 1. Federated Learning Background 2. Brief Introduction of FATE/KubeFATE 2. FATE on Pulsar: Design and Architecture 1. FATE’s architecture 2. Why Apache Pulsar 3. Details of the Implementation 3. Summary
- 5. ©2020 VMware, Inc. 5 What is Federated Learning? • 2+ organizations co-train a model; • No raw data exchange (privacy-preserving); • Encrypted model trained and averaging: • Homomorphic encryption • Secret-sharing • Oblivious transfer • …
- 6. ©2020 VMware, Inc. 6 What is Federated Learning? • 2+ organizations co-train a model; • No raw data exchange (privacy-preserving); • Encrypted model trained and averaging: • Homomorphic encryption • Secret-sharing • Oblivious transfer • … Traditional machine learning Federated learning (Source: Federated Learning (Synthesis Lectures on Artificial Intelligence and Machine Learning) , Qiang yang , et al. ）
- 7. ©2020 VMware, Inc. 7 Catalogs of Federated Learning Data B Horizontal Federated Learning Vertical Federated Learning Data A Data B Label Data A Sample Sample Features Label Features Situation 1: Samples are not enough, distributed and cannot be gathered because privacy restrictions… Situation 2: Features of one sample is distributed in different organizations… Sample-partitioned Federated Learning (Source: https://ai.googleblog.com/2017/04/federated-learning-collaborative.html ） (Source: Chinese idioms: blind men touch an elephant） Feature-partitioned Federated Learning
- 8. ©2020 VMware, Inc. 8 Use cases of Federated Learning Data B Horizontal Federated Learning Vertical Federated Learning Data A Data B Label Data A Sample Sample Features Label Features (Source: https://www.fedai.org/cases/utilization-of-fate-in-risk-management-of-credit-in-small-and-micro-enterprises/ ） Risk Management of Credit in Small and Micro Enterprises Full user images with features from different areas AUC improve 12% Anti Money Laundering Through Multiple Banks Small silo-dataset to big data under compliance enforcement (Source: https://www.fedai.org/cases/utilization-of-fate-in-anti-money-laundering-through-multiple-banks/ ） Encrypted Model Inferring query Calculate inferring values Get result Webank anti-money laundry platform Download encrypted model at the first time Query probability with calculated value Return probability AUC improve 14%
- 9. ©2020 VMware, Inc. 9 FATE: Federated AI Technology Enabler ● An open-source project hosted by Linux Foundation ■ 3000+ stars, 7000+ commits, 38 contributors from different organizations ● Industry level federated learning framework with out-of-box usability ● Enable data collaboration with data protection regulation compliance ● Support both sample-partitioned and feature- partitioned federated machine learning The only opensource and supporting feature-partitioned federated learning solution by now. https://github.com/FederatedAI/FATE
- 10. ©2020 VMware, Inc. 10 Current Algorithms in FATE
- 11. ©2020 VMware, Inc. 11 KubeFATE manages federated learning workloads using cloud native technologies such as containers, which enables federated learning jobs to run across public, private and hybrid cloud environments. https://github.com/FederatedAI/KubeFATE KubeFATE: Accelerate the FL with Cloud Natvie Technologies Containers KubeFATE
- 12. ©2020 VMware, Inc. FATE on Pulsar The Design and Architecture
- 13. ©2020 VMware, Inc. Communications among FL Participants Organization A Organization B Organization C 1. Multiple transmission within one iteration 2. Cross silo 3. 30 times of message size after encryption Server A Server B Server C Distributed Learning Federated Learning
- 14. ©2020 VMware, Inc. Why Message Queue Data Data Data Data Data Data Executor Mirroring FATE cluster of organization A MQ MQ The advantages of mirroring 1. Stable 2. Support big data scenarios 3. Design for WAN FATE cluster of organization B
- 15. ©2020 VMware, Inc. The Services of a FATE Cluster FATE Board Mysql FATE Flow HDFS Spark Master Spark Worker Spark Worker Message Queue Nginx Message Queue Nginx Data Flow FATE Party B FATE Party A … …
- 16. ©2020 VMware, Inc. The key features of Pulsar： • Geo-replication • Message compression • Low latency • Cloud native architecture • Support massive topics • Support multiple clients Why Pulsar The compose of a Pulsar cluster： • One or more brokers for processing and load balancing messages; • A BookKeeper cluster to persist the messages. • A Zookeeper cluster to store metadata or to coordinate tasks between multiple Pulsar clusters.
- 17. ©2020 VMware, Inc. The Geo-replication Feature of Pulsar Topic A-B-C Topic A-B Topic A-B-C Topic A-B Topic A-B-C Pulsar cluster A Pulsar cluster B Pulsar cluster C Consumer C1 Producer P1 Consumer C2 Producer P2 Consumer C3 Consumer C4
- 18. ©2020 VMware, Inc. The Details of FATE on Spark with Pulsar Topic Topic Topic Topic Topic Topic partition partition partition Executor Executor Executor partition partition partition Executor Executor Executor FATE Flow Driver FATE Flow Driver Nginx Geo-replication Nginx Party A Party B Pulsar broker Pulsar broker
- 19. ©2020 VMware, Inc. FATE can implement star networking when using Pulsar as a transmission service. It uses an SNI (Server Name Indication) proxy server as the center. The specific proxy service can be Apache Traffic Server. A three-party star network architecture is as follows: The Star Network ATS Pulsar Fate Flow Spark Cluster Party 9999 192.168.0.3 … Nginx Pulsar Fate Flow Spark Cluster Party 7777 Nginx Pulsar Fate Flow Spark Cluster Party 8888 … Nginx Nginx Exchange exchange.fate.org 7777.fate.org 8888.fate.org 9999.fate.org …
- 20. ©2020 VMware, Inc. The Details of SNI Proxying ATS Pulsar Cluster B Pulsar Cluster A 7777.fate.org 9999.fate.org exchange.fate.org 1. TLS Client hello { sni: 9999.fate.org: 6651 } 3. TLS Server hello 2. TCP Tunnel SNI route table The SNI agent process is as follows: • The client sends a TLS client Hello request to the proxy server, with an SNI field in the request, which declares the domain name or host name the client wants to connect to. • The proxy server establishes a TCP tunnel with the remote server according to the SNI and the route table, then it forwards the message to server. • The remote server sends a TLS Server Hello to the client and then completes the TLS handshake. The TCP link is established, and the client and the remote server can perform further communication. sni: - fqdn: 9999.fate.org tunnel_route: 192.168.0.2:6651 - fqdn: 7777.fate.org tunnel_route: 192.168.0.3:6651
- 22. ©2020 VMware, Inc. 22 1. Federated learning is data and network intensive workload; 2. Pulsar is designed for cloud native, supports start networking with SNI proxy; 3. Pulsar provides more stable and throughput comparing to other MQ like RabbitMQ, etc. 4. Pulsar provides multiple subscription modes and message compression to improve messaging performance. Summary
- 23. ©2020 VMware, Inc. 23 1. Continue to tune FATE on Pulsar 2. Use Pulsar for ETL, job scheduling, etc. in FATE 3. Support Pulsar's cluster mode deployment in KubeFATE Future Works