Containerising the Mule Runtime with Kubernetes & From Zero to Batch : MuleSoft Virtual Muleys Meetups
2 likes1,123 views
The document provides details about a MuleSoft meetup discussing containerizing the Mule Runtime with Kubernetes, including logistics for the event and a summary of the topics covered such as CI/CD, deployment strategies, and batch processing concepts. It also includes announcements about upcoming events, training certifications, and updates on features like multi-factor authentication and API Designer enhancements. Additionally, essential guidelines for community engagement and accessing support resources are provided.
Containerising the Mule Runtime with Kubernetes & From Zero to Batch : MuleSoft Virtual Muleys Meetups
- 1. All contents © MuleSoft, LLC From zero to Batch Anu Vijayamohan Containerizing the Mule Runtime with Kubernetes Arno Brugman 18th November 17:00 London (GMT) | 18:00 Europe | 22:30 India (Delhi) 9:00 San Francisco | 12:00 New York
- 2. All contents © MuleSoft, LLC MuleSoft Meetups: Virtual Muleys (Online Group - English ) November, 2020Recording
- 3. All contents © MuleSoft, LLC MuleSoft Meetups: Virtual Muleys 3
- 4. All contents © MuleSoft, LLC Keep in mind! 1. For Trainings and Certifications - https://training.mulesoft.com/ ○ Still NEW: Mule 4 MCD FREE Certification voucher when completing the Mule 4 Self-Serve course! 2. For Training queries: https://help.learn.mulesoft.com/hc/en-us/requests/new 3. Doubts? ○ We have a very good community to help you! https://help.mulesoft.com/ ○ Or leverage StackOverflow (https://stackoverflow.com/) ( specially for those working on Dataweave and Runtime) 4
- 5. All contents © MuleSoft, LLC Guidelines for today 1. Pop questions in the chat and anyone can try to answer! 2. Remaining questions will be asked in your behalf at the end of the session 3. The Recording will be shared in the Meetups page and in the VirtualMuleys Youtube Channel 4. Your feedback drives the content of these meetups, fill the surveys at the end of the event, let us know what you think! 5
- 6. All contents © MuleSoft, LLC Announcements and Updates
- 7. All contents © MuleSoft, LLC Start playing with DW 2.0 from your browser: http://dwlang.fun/ Pre-Alpha: DataWeave Playground & Tutorial 7 Disclaimer: It is not an officially supported MuleSoft Product. It is not supported by MuleSoft Support, you won't be able to open MuleSoft Support cases for this Suite and there are no SLAs for responses nor bug/features progress.
- 8. All contents © MuleSoft, LLC Start playing at: http://dwlang.fun/ Pre-Alpha: DataWeave Playground & Tutorial 8 Disclaimer: It is not an officially supported MuleSoft Product. It is not supported by MuleSoft Support, you won't be able to open MuleSoft Support cases for this Suite and there are no SLAs for responses nor bug/features progress.
- 9. All contents © MuleSoft, LLC ● Repo and Pull Requests: https://github.com/mulesoft-labs/data-weave-tutorial “Pre-Alpha” - DataWeave Playground & Tutorial 9
- 10. All contents © MuleSoft, LLC Compete in the first virtual MuleSoft Hackathon! ● You have until the Nov. 29, 2020 to submit! ● Amazing prices with lot of fun ● For the event: https://www.mulesoft.com/hackathon2020 ● Recording with all the details: https://meetups.mulesoft.com/mulesoft-hackatho n-2020/ ● Disclaimer: Country and additional eligibility restrictions will apply.
- 11. All contents © MuleSoft, LLC Available Now
- 12. All contents © MuleSoft, LLC Marina Rivero Anypoint Studio 7.7 deep dive with MuleSoft engineering! Eric Mastrangelo 25th November 17:00 London (GMT) | 18:00 Europe | 22:30 India (Delhi) 9:00 San Francisco | 12:00 New York https://meetups.mulesoft.com/e/mphrn6/
- 13. All contents © MuleSoft, LLC ● 5-15 minutes videos with a lot of good content. ● Introduction: Continuous Integration and Continuous Deployment (CICD) ● Overview (Why, what is it) and how you can articulate the value of CICD to different Stakeholders ● Find it here: https://sfdc.co/cokrDc Friends of Max Videos 13 Build PassTest BuildTest Pass Automation Server Automated Automated
- 14. All contents © MuleSoft, LLC Simple and effective ways to protect your data Secure your user accounts with multi-factor authentication Something you know Login Credentials Something you have MFA Find more information in: https://docs.mulesoft.com/access-management/multi-factor-authentication
- 15. All contents © MuleSoft, LLC Secure and strong verification methods Verification methods supported by MuleSoft SMS (Text) verification Phone call verification Email verification Not Supported ( x x Salesforce Authenticator Mobile App Third-Party Authenticator Apps Security Keys Built-In Authenticators Such as: Google Authenticator Microsoft Authenticator Authy Such as: Yubico’s YubiKey Google’s Titan Security Key Desktop + mobile device biometrics, such as: Windows Hello Touch ID Face ID Fast, free authentication sfdc.co/IntrotoAuthenticator
- 16. All contents © MuleSoft, LLC ● Multi-tabbing has been a highly requested feature from our Ideas Portal ● We are excited to announce that it has been added to API Designer to improve your API specification development experience. ● Whether you have one RAML file or 12, simply CMD + click (Mac OS) or CTRL + click (Windows) on a file in the left-hand panel to open a new tab in API Designer 16 Available Now https://blogs.mulesoft.com/dev/design-dev/how-to-use-multi-tabbing -and-rename-symbol/ Take advantage of the Multi-tabbing Feature
- 17. All contents © MuleSoft, LLC From zero to Batch Anu Vijayamohan Containerizing the Mule Runtime with Kubernetes Arno Brugman 18th November 17:00 London (GMT) | 18:00 Europe | 22:30 India (Delhi) 9:00 San Francisco | 12:00 New York
- 18. All contents © MuleSoft, LLC Arno A. Brugman, Strategic Advisor Containerizing the Mule Runtime with K8s MuleSoft Meetup Wed, Nov 18, 5:00 PM (GMT)
- 19. All contents © MuleSoft, LLC Safe harbor statement The information in this presentation is confidential and proprietary to MuleSoft and may not be disclosed without the permission of MuleSoft. This presentation is not subject to your license agreement or any other service or subscription agreement with MuleSoft. MuleSoft has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation and MuleSoft's strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by MuleSoft at any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This document is for informational purposes and may not be incorporated into a contract. MuleSoft assumes no responsibility for errors or omissions in this document, except if such damages were caused by MuleSoft intentionally or grossly negligent. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.
- 20. All contents © MuleSoft, LLC About the Speaker 20 Arno Brugman ● 30+ years in IT ● 4 years with MuleSoft ● Strategic Advisor ○ EMEA’s most ambitious customers ○ Trusted Advisor ○ Product & Technology ○ Customer Advocate
- 21. All contents © MuleSoft, LLC Agenda 1. Introduction to Containerization 2. Containerization Environments 3. Containerizing the Mule Runtime with Kubernetes 4. Building the Image 5. Deploying the application 6. CI/CD 7. Runtime Fabric 21
- 22. All contents © MuleSoft, LLC Introduction to Containerization
- 23. All contents © MuleSoft, LLC Introduction to Containerization A brief history ● MULTICS (1963) ● UNICS (1969) ● MVS/zOS (1974) ● VAX/VMS (1977) ● Chroot Jail (1979) ● Windows (1983) ● Linux (1991) ● VMWare Workstation (1999) ● Linux Containers (2008) ○ LXC ● docker (2013) ○ Based on LXC 23
- 24. All contents © MuleSoft, LLC docker definition of Containers Introduction to Containerization A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another 24
- 25. All contents © MuleSoft, LLC docker definition of Containers (2) Introduction to Containerization A container is an abstraction at the application layer that packages code and dependencies together. Multiple containers can run on the same machine and share the OS kernel with other containers, each running as isolated processes in userspace. Containers take up less space than VMs, can handle more applications, and require fewer VMs and Operating systems 25
- 26. All contents © MuleSoft, LLC Vs Virtualization Containers 26 Infrastructure (Host) Operating System docker App1 App2 App3 App4 App5 App6 App7 App8 App9 Infrastructure Hypervisor Virtual Machine Virtual Machine Virtual Machine Virtual Machine Guest OS Guest OS Guest OS Guest OS App 1 App 2 App 3 App 4
- 27. All contents © MuleSoft, LLC Containerization Environments
- 28. All contents © MuleSoft, LLC Containerization Environments ● Alibaba: Container Service for Kubernetes + Container Service ● Amazon: ECS + EKS ● Canonical ● Google: GKE ● Gravitational: Gravity ● Heptio ● IBM: Cloud Kubernetes Service + Multicloud Manager ● Kontena Pharos ● Microsoft: AKS + ACI ● Oracle: Application Container Cloud ● Pivotal: Cloud Foundry + Container Service ● Rancher ● RedHat: OpenShift ● Suse 28
- 29. All contents © MuleSoft, LLC Most Commonly used Containerization Environments ● Cloud Platforms ○ Amazon EKS ○ Google GKE ○ Microsoft AKS 29 ● On Premise ○ RedHat OpenShift ○ Pivotal Cloud Foundry
- 30. All contents © MuleSoft, LLC Containerizing the Mule Runtime with Kubernetes
- 31. All contents © MuleSoft, LLC Containerizing the Mule Runtime with Kubernetes Pods vs Standalone Server 31 Kubernetes Server Mule App Standalone Mule Runtime App App App App App App ... Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App
- 32. All contents © MuleSoft, LLC Advantages vs Standalone Containerizing the Mule Runtime with Kubernetes ● Application Isolation ○ Dedicated CPU ○ Dedicated Memory ○ Dedicated Disk ● Configurable ○ CPU, Memory, Disk ● Horizontal Scaling ● Vertical Scaling 32
- 33. All contents © MuleSoft, LLC However... Containerizing the Mule Runtime with Kubernetes ● Application Isolation ○ Dedicated CPU ○ Dedicated Memory ○ Dedicated Disk ● Configurable ○ CPU, Memory, Disk ● Horizontal Scaling ● Vertical Scaling 33 ● No Automatic Load Balancing ● No Zero Downtime re-deploy ● Build your own Image ● Apply Patches yourself
- 34. All contents © MuleSoft, LLC However... Containerizing the Mule Runtime with Kubernetes 34 Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Pod Mule App Proxy Anypoint Platform
- 35. All contents © MuleSoft, LLC Building the Image
- 36. All contents © MuleSoft, LLC Base Image Building the Image ● OS ● JDK ● Mule Runtime ● License File 36
- 37. All contents © MuleSoft, LLC Hardware ● 2 GHz CPU (or 1 virtual CPU in virtualized environments) ● 1 GB of RAM ● 4 GB of storage Software ● RHEL 7 ● JDK 1.8.0 or JDK 11 ● Mule Runtime version 4.3 ● Mule License File Building the Image What do you need (for Mule 4.3)? 37
- 38. All contents © MuleSoft, LLC Deploying the Application
- 39. All contents © MuleSoft, LLC Deploying the application Method 1: Unmanaged ● Copy the Application.zip fille into the container ● Run mule 39 ● Not Managed by Anypoint runtime Manager
- 40. All contents © MuleSoft, LLC Method 2: Managed by Anypoint Runtime Manager Deploying the application ● Install the Runtime Manager Agent ● Upload your application (CLI/API) ● Deploy your application into the container 40
- 41. All contents © MuleSoft, LLC CI/CD
- 42. All contents © MuleSoft, LLC CI/CD Startup After starting the Mule Runtime, the Runtime needs to be connected to the Anypoint Platform. Use the Anypoint Platform API to register the runtime 42 Shutdown When a container is shutdown, the container instance (Name + IP) is terminated. Before Shutdown, the container needs to be properly de-registered from the Anypoint Platform using the Anypoint Platform API
- 43. All contents © MuleSoft, LLC Runtime Fabric
- 44. All contents © MuleSoft, LLC Anypoint Runtime Fabric Deploy and manage runtimes across any cloud Centrally manage all applications Manage applications in your private cloud (AWS, Azure) and CloudHub within Anypoint Platform Run Mule apps on Kubernetes Get containerization benefits such as isolation, horizontal scaling, and auto redeploy by default Move deployments in a few clicks Pivot your deployments between CloudHub, your private cloud, or a data center with no impact
- 45. All contents © MuleSoft, LLC Anypoint Runtime Fabric VM Mule App VM Mule App Mule App Runtime Fabric components Runtime Fabric appliance Mule App network Runtime Fabric Mule App Mule App VM Runtime Fabric orchestrates and automates the deployment of Mule runtimes into containers in any cloud or on-premises environment Benefits ● Deploy consistently across any cloud or data center ● Run multiple runtime versions in the same Runtime Fabric ● Scale horizontally and redeploy w/ zero-downtime ● Easily manage via the control plane hosted by MuleSoft ● Flexible deployment upon existing infrastructure or managed K8s services
- 46. All contents © MuleSoft, LLC Anypoint Runtime Fabric on EKS and AKS
- 47. All contents © MuleSoft, LLC Anypoint Runtime Fabric on EKS and AKS Available now Available NOW EKS AKS 47
- 48. All contents © MuleSoft, LLC Deploy and manage runtimes on AKS or EKS Runtime Fabric on AKS and EKS Extend control plane benefits to runtimes on your AKS or EKS ● Customer gets the latest management and monitoring features of Anypoint Platform Decouple Mule runtime services from your own infrastructure ● Give your ops teams the power to use their infrastructure how they want48 CUSTOMER’S NETWORK Mule App Mule App Mule App Runtime Fabric services Mule App Mule App Mule App Mule App Mule App Mule App Runtime Plane CONTROL PLANE Kubernetes-as-a-Service Available now
- 49. All contents © MuleSoft, LLC Deploy and manage across any K8s service Deploy across private clouds Open K8s access to managed providers Expanded K8s access on managed provider Across any K8s service EKS AKS GKE ● Deploy standalone cluster within AWS, Azure, or on-prem ● Use K8s and docker without K8s expertise ● Install RTF services into private K8s environment ● Open access for customers with existing K8s expertise ● Support for OpenShift container platform ● Run Mule and non-Mule workloads on the same K8s environment ● Open support for any managed service built on K8s ● Deploy across any K8s environment
- 50. All contents © MuleSoft, LLC 50 Thank you
- 51. All contents © MuleSoft, LLC Anu Vijayamohan Customer Success Technical Architect, EMEA From zero to Batch MuleSoft Meetup: 18 November 2020
- 52. All contents © MuleSoft, LLC About the Speaker 52 Anu Vijayamohan, Customer Success Technical Architect ● American born, based in the UK ● Almost 20 years of field expertise ● Focussed mostly in working directly in IT Delivery and Professional Services ● Worked for Appian for more than 13 years ● Joined MuleSoft Customer Success about 1.5 years ago
- 53. All contents © MuleSoft, LLC Agenda 1. Introduction to Batch Processing 2. Batch Processing Use Cases 3. Batch Processing Features in Mule 4 4. Best Practices for Batch Processing 5. Error Handling 6. Performance Tuning Considerations 7. Certification Tips 53
- 54. All contents © MuleSoft, LLC Introduction to Batch Processing
- 55. All contents © MuleSoft, LLC restaurant 55
- 56. All contents © MuleSoft, LLC Batch Processing Use Cases
- 57. All contents © MuleSoft, LLC Batch Processing Scenarios Extracting, transforming and loading (ETL) information into a target system Handling large quantities of incoming data from an API into a legacy system Integrating data sets to parallel process records Synchronising data sets between business applications 57
- 58. All contents © MuleSoft, LLC API vs Batch Characteristics 58 Aspect API Batch Availability On-demand based on user interaction (e.g. real-time) Scheduled copy of data-set that doesn’t require user interaction Profile • Definition: configurable fields/attributes, join, filter, sort, etc. • Volume: control data-set size e.g. 10,000 records per request • Type: transactional, status, etc. • Self-service: empowerment • Definition: static or changes infrequently • Volume: Payload size < 2GB, million+ records, Big Data • Type: historical, reporting/analytics, master Data Change Frequency Real-time, near real-time, minutes Daily, weekly, monthly Consumer One or more Destination/s Publication Event oriented N/A
- 59. All contents © MuleSoft, LLC Batch Processing Features in Mule 4
- 60. All contents © MuleSoft, LLC Batch Job Scope in Mule 4 60
- 61. All contents © MuleSoft, LLC Batch Processing Phases 61
- 62. All contents © MuleSoft, LLC Batch Processing Phases 62
- 63. All contents © MuleSoft, LLC Batch Flow Load Batch Step Error Handling Bulk Insert Logging 63
- 64. All contents © MuleSoft, LLC Benefits of Using the Batch Job Scope 64 ● Data Integrity ● Reliability ● Efficiency
- 65. All contents © MuleSoft, LLC 65 ● Polling Connectors ● Poll Scope ● For-Each ● Watermarks Batch and ETL Using Mule Flows Sequential Record Processing Capabilities
- 66. All contents © MuleSoft, LLC Best Practices for Batch Processing
- 67. All contents © MuleSoft, LLC Mule 4 Batch Processing Basics ● Mule processes a batch job asynchronously ● Any event source that feeds data into a batch job must be one-way, not request-response ● Results of batch processing do not feed back to the caller ● Default settings ○ Process records in blocks of 100 (Batch Block Size) ○ 16 thread default threading profile ○ Cancel processing on record failure (Max Failed Records; default is 0) ● Check out the Mule documentation for more detail 67
- 68. All contents © MuleSoft, LLC Batch Processing Strategies ● Pattern strategy ● Exception strategy ● Capture and design around NFRs 68
- 69. All contents © MuleSoft, LLC Running Batch Processes ● Front load transformation ● Separation of deployments ● Co-running multiple batch jobs ● File sizes ● Watermarking 69
- 70. All contents © MuleSoft, LLC Streaming Best Practices ● Use streaming batch processing when writing to a file ● Performance implications ● Item access implications 70
- 71. All contents © MuleSoft, LLC Error Handling
- 72. All contents © MuleSoft, LLC Error Logging Mule 4 Default When a batch step fails or raises an error, Mule logs the stack traces following this logic: ● Mule gets the exception’s full stack trace. ● Mule strips that stack trace from any messages. ● Mule verifies if the stack trace was already logged in the current step ○ If this was the first time the runtime encountered this error, it logs the error and showing a message like this: com.mulesoft.mule.runtime.module.batch.internal.DefaultBatchStep: Found exception processing record on step 'batchStep1' for job instance 'Batch Job Example' of job 'CreateLeadsBatch'. ○ Subsequent records with the same failures will not be logged for performance and log readability reasons: ● Mule logs on a "by step" basis. If another step also raises the same error, the runtime logs it again for that step 72
- 73. All contents © MuleSoft, LLC Error Logging Mule 4 Default ● When the batch job reaches the On Complete phase, Mule displays an error summary with every error type, and how many times it happened in each batch step. ● Mule logs are only logs INFO level messages by default ● For more verbose logs, set the batch category to DEBUG level in the application’s log4j configuration 73
- 74. All contents © MuleSoft, LLC Error Handling at a Record Level ● Finish Processing ● Continue Processing Configurable Options 74
- 75. All contents © MuleSoft, LLC Performance Tuning Considerations
- 76. All contents © MuleSoft, LLC Performance Tuning Considerations 76 ● Focus on each process step to look for specific code optimizations ● Focus on transformation and any external connectivity logic within process steps ● Eliminate as much disk and network latency as possible ● Align performance tuning considerations with current customer performance testing activities ● Consider Disk I/O
- 77. All contents © MuleSoft, LLC Certification Tips
- 78. All contents © MuleSoft, LLC MuleSoft Certified Integration Architect (MCIA) Batch Topics Covered in Exam ● ForEach scope ● Batch Job scope ○ Synchronicity ○ Filters ○ Pros/Cons ● Bulk Operations in connectors ● Streaming ● Understand the concepts, don’t just memorize ● Learn by doing! Open up Anypoint Studio and debug code to understand the behaviour ● Exam questions require a problem to be solved by weighing different options 78
- 79. All contents © MuleSoft, LLC Thank you