Mule soft meetup_chandigarh_#7_25_sept_2021
- 1. 25 September 2021 Chandigarh MuleSoft Meetup Group Snowflake and Apache Kafka with Mulesoft
- 2. 2 Guidelines for our Virtual Meetup 1. You can write down your questions in the chat during the session. 2. Once we are done with the day, don’t forget to provide your valuable feedback.
- 3. 3 ●About the organizers: ○ Suresh Rathore ○ Lalit Panwar ○ Ajay Rana Introductions A SHOW OF HANDS: Who is new to this Meetup? ●About the sponsor: ○ MuleSoft
- 5. 5 ● Event driven architecture overview ● What is Snowflake ? ● Feature and benefits of Snowflake ● Snowflake Integration with Mulesoft ● Apache Kafka introduction ● Integration with Mulesoft ● Lets see how they fits together in Mulesoft. Agenda
- 6. 6
- 8. 8 ● It’s a 100% cloud-based database and data warehouse ● Benefits of Snowflake: ○ There is no hardware (virtual or physical) to select, install, configure, or manage. ○ There is virtually no software to install, configure, or manage. ○ Ongoing maintenance, management, upgrades, and tuning are handled by Snowflake. Introduction
- 9. 9
- 10. 10
- 11. Integrate Snowflake with mulesoft ● One can sign up on Snowflake for trial using below link ○ https://trial.snowflake.com/ ● Select Snowflake edition as Enterprise and Cloud Provider like Amazon Web Services (AWS). 11
- 12. APIs ● For activation an email will sent to user account. ● After successfully logged in, User can create their own databases and schemas. ● In Anypoint studio search Snowflake in exchange and add Snowflake Connector. 12
- 14. Apache Kafka
- 15. Introduction ● Its an open-source stream-processing software platform ● It can handle about trillions of data events in a day. ● Apache Kafka creates real-time streaming apps and data pipelines. ● It is scalable, fault-tolerant, and incredibly fast. 15
- 16. Core API’s 16 ● Producer API: This API allows/permits an application to publish streams of records to one or more topics. ● Consumer API: This API allows an application to subscribe one or more topics and process the stream of records produced to them..
- 17. Core API’s 17 ● Streams API: This API allows an application to effectively transform the input streams to the output streams. It permits an application to act as a stream processor which consumes an input stream from one or more topics, and produce an output stream to one or more output topics. ● Connector API: This API executes the reusable producer and consumer APIs with the existing data systems or applications.
- 19. 19
- 20. 20 How they fits together?
- 21. Steps for Apache Kafka 1. Download Apache Kafka binary file from https://kafka.apache.org/downloads 2. Set environment variable for kafka in path as : “Kafka folder>bin>windows” 3. Check “kafka--” in command prompt to check if its installed or not. 4. Set Kafka-logs path in ZooKeeper and Kafka config files respectively. 5. Following commands are used to start and create topics in Kafka : ○ zookeeper-server-start.bat configzookeeper.properties ○ kafka-server-start.bat configserver.properties ○ kafka-topics --zookeeper localhost:2181 --topic myfirst --create --partitions 3 --replication-factor 1 ○ kafka-topics.bat --zookeeper localhost:2181 --list ○ kafka-topics.bat --zookeeper localhost:2181 --topic myfirst --delete 21
- 22. Integration steps with Mulesoft ● In Studio, Click the Exchange (X) icon in the task bar. ● Select All assets and search for Apache Kafka. ● Select the connector and click Add to project. 22
- 23. Why Kafka uses Zookeeper ? ● ZooKeeper is used in distributed systems for service synchronization ● In Apache Kafka, ZooKeeper is primarily used to track the status of nodes in the Kafka cluster and maintain a list of Kafka topics and messages. 23
- 25. Q&A
- 27. Further Resources ● Links : • https://www.royalcyber.com/blog/apache-kafka/apache-kafka-integration-with-mulesoft/ • https://apisero.com/about-snowflake-and-its-integration-with-mulesoft/ ● Mule 4 Connector: • https://docs.mulesoft.com/snowflake-connector/1.0/snowflake-connector-studio • https://docs.mulesoft.com/kafka-connector/4.5/kafka-connector-studio
- 28. Trivia Questions
- 29. 29 ● Share: ○ Tweet your pictures or posts using the hashtag #MuleSoftMeetups ○ Invite your network to join: https://meetups.mulesoft.com/chandigarh/ ● Feedback: ○ Fill out the survey feedback and suggest topics for upcoming events ○ Contact MuleSoft at [email protected] for ways to improve the program It always seems impossible until it is done - NELSON MANDELA What’s next?
- 30. Thank you