![All contents © MuleSoft Inc.
Reading Data
21
• In Last slide we have seen methods of creating the data, now we will see
how we can read data.
• Dataweave selector will allow navigating any combination of Objects and
Arrays to get the data we need.
• Some of the selectors we will be looking into are:
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
Selector Representation Example
Single Value Selector . Payload.name
Index Selector [n] Payload[1]
Range Selector [n to m] Payload[0..2]
Multi Value Selector * Payload.*name
Descendants Selector .. Payload..echo](https://image.slidesharecdn.com/mulesoftmeetupcharlotte4draftv2-211117185431/85/Mule-soft-meetup_charlotte_4__draft_v2-0-21-320.jpg)
![All contents © MuleSoft Inc.
Map
29
• map(Array<T>, (item: T, index: Number) -> R): Array<R>
• map contains key value pairs
• Example:
%dw 2.0
output application/json
---
["jose", "pedro", "mateo"] map (value, index) -> { (index) : value}
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.](https://image.slidesharecdn.com/mulesoftmeetupcharlotte4draftv2-211117185431/85/Mule-soft-meetup_charlotte_4__draft_v2-0-29-320.jpg)
![All contents © MuleSoft Inc.
Distinct By
30
• Iterates over an array and returns the unique elements in it.
• Other versions act on an object and handle a null value
• DataWeave uses the result of applying the provided lambda as the
uniqueness criteria.
%dw 2.0
output application/json
---
[0, 1, 2, 3, 3, 2, 1, 4] distinctBy (value) -> { "unique" : value }
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.](https://image.slidesharecdn.com/mulesoftmeetupcharlotte4draftv2-211117185431/85/Mule-soft-meetup_charlotte_4__draft_v2-0-30-320.jpg)
![All contents © MuleSoft Inc.
Group By
31
• Returns an object that groups items from an array based on specified
criteria, such as an expression or matching selector.
• This version of groupBy groups the elements of an array using the criteria
function. Other versions act on objects and handle null values.
• Example:
%dw 2.0
var myArray = [
{ "name": "Foo", "language": "Java" },
{ "name": "Bar", "language": "Scala" },
{ "name": "FooBar", "language": "Java" }
]
output application/json
---
myArray groupBy (item) -> item.language
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.](https://image.slidesharecdn.com/mulesoftmeetupcharlotte4draftv2-211117185431/85/Mule-soft-meetup_charlotte_4__draft_v2-0-31-320.jpg)
![All contents © MuleSoft Inc.
Reduce
32
• Applies a reduction expression to the elements in an array.
• or each element of the input array, in order, reduce applies the reduction
lambda expression (function), then replaces the accumulator with the new
result.
• The lambda expression can use both the current input array element and
the current accumulator value.
• Example:
%dw 2.0 output application/json --- [2, 3] reduce ($ + $$)
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.](https://image.slidesharecdn.com/mulesoftmeetupcharlotte4draftv2-211117185431/85/Mule-soft-meetup_charlotte_4__draft_v2-0-32-320.jpg)
Mule soft meetup_charlotte_4__draft_v2.0
- 1. All contents © MuleSoft Inc. Welcome To MuleSoftMeetup 1
- 2. All contents © MuleSoft Inc. Welcome To MuleSoftMeetup 2 MuleSoft Meetup #4 Virtual Nov 16, 2021
- 3. All contents © MuleSoft Inc. Agenda 3 6:30 PM Welcome & Introductions 6:45 PM Exploring ETL use cases for Salesforce as target system using MuleSoft's Bulk API connectors and batch processing by Amresh Kosuru 7:15 PM Deep Dive into Dataweave 2.x and its Modules by Aravind Babu Ramadugu 7:45 PM Q/A, Trivia & Event Close
- 4. All contents © MuleSoft Inc. Announcements Hi Everyone, Thanks for joining our 4th Virtual Charlotte Meetup today. Today we are going to have talks about Exploring ETL use cases for Salesforce and Deep Dive into Dataweave 2.x and its Modules . At the end, we will have Trivia session, where we will ask some random questions related to the topics which we discussed today. First 3 winners will get a free training and certification voucher from MuleSoft. Do not miss the opportunity…! Without further ado, let’s get started.
- 5. ETL with Salesforce Bulk API V2
- 6. All contents © MuleSoft Inc. Agenda 6 - ETL use cases – Extract Transform and Load in context of Salesforce. - Salesforce APIs - a brief intro. - MuleSoft Salesforce Bulk API V2 connector. - Crash course on Batch processing in Mule. - Demo - Limits and Tuning - Conclusion
- 7. All contents © MuleSoft Inc. ETL Tools - Vs - Mule ESB as an ETL tool 7 Specialized for hauling Heavier Hauls Costs more for gas Need some skill with driving big vehicles Needs wider and longer parking space Best of both worlds Light to moderate Hauls Costs less for gas Easy driving Just the regular parking lot Just for Fun!! – a comparison!!!
- 8. All contents © MuleSoft Inc. Salesforce 8 What is Salesforce? – CRM – Customer Relationship Management – SaaS platform for managing all things related to Customer relationship. – Cloud based – An online database of business objects in context of CRM Contact Account Opportunity Activity Campaign Lead
- 9. All contents © MuleSoft Inc. Use Cases – C.R.U.D 9 SQL Files Staging Databases Legacy Apps Partnering Apps Full Loads Delta Loads Data Syncing Insert Upsert Delete Query
- 10. All contents © MuleSoft Inc. Salesforce Platform APIs 10 SOAP APIs REST APIs Streaming APIs APEX APIs BULK APIs (V2) • A specialized Restful API for larger data loads. • 2000 records or more. • Asynchronous. • V2 – easier to use than V1. • Ideal for ETL use cases. • Batch processing in the background. • Automatic Optimization for best possible performance. • Processes records, in batches, under the scope of a Job.
- 11. All contents © MuleSoft Inc. Salesforce Platform APIs - Continued 11 Bulk API V2 Life Cycle
- 12. All contents © MuleSoft Inc. Ingest Mule Salesforce Connector – Needed operations 12 Query
- 13. All contents © MuleSoft Inc. Demo 13
- 14. Dataweave 2.x
- 15. All contents © MuleSoft Inc. Agenda 15 • Introduction • Creating Data • Reading Data • Variables & Logical Operators • Flow Control • Functions • Working With Arrays • Working With Objects
- 16. All contents © MuleSoft Inc. Introduction 16 • Dataweave is Language designed for transforming data • MuleSoft primary language for data transformation • Also available in other contexts like command line tool • Every system out there wants to be special and define its own data format • Dataweave provides a simple JSON-like functional programming language to help quickly transform and query data.
- 17. All contents © MuleSoft Inc. Advantages of Dataweave 17 • Easy to write, easy to maintain, and capable of supporting simple to complex mappings between any data types – Out of the box support for many popular file and data types • XML, JSON, Java, CSV, EDI, Excel, fixed-width files, flat files, and COBOL copybook • Often more succinct than custom code – Do not require any intermediate Java class or annotations • Reusable and more readable by front-end developers – Data transformations can be stored in external DWL files (modules) and used across applications – Powerful functional programming using JavaScript like syntax
- 18. All contents © MuleSoft Inc. Dataweave mechanics 18 • Internally, Dataweave includes a connectivity layer and an engine that is fundamentally different from other transformation technologies. • It contains a data access layer that indexes content and accesses the binary directly, without costly conversions – Enables larger than memory payloads – Random access to input documents, even larger than memory – Performance intensive
- 19. All contents © MuleSoft Inc. Creating Data 19 • Data Type: – Strings, Booleans, Numbers, Objects, Arrays • We can check the type of the data by using following command. – typeOf: Will give the type associated to the sample data. • Strings: – Strings are defined between quotes • Numbers – Supports number type covering both Integers and NUmbers • Booleans – his is the last simple type and has values of either true or false. • Arrays – Arrays are an ordered series of values where the values can be of any type: • Objects – Essentially Key Value Pairs – Values can be of any data type DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 20. All contents © MuleSoft Inc. Creating Data 20 • Data Type: – Strings, Booleans, Numbers, Objects, Arrays • We can check the type of the data by using following command. – typeOf: Will give the type associated to the sample data. • Strings: – Strings are defined between quotes • Numbers – Supports number type covering both Integers and NUmbers • Booleans – his is the last simple type and has values of either true or false. • Arrays – Arrays are an ordered series of values where the values can be of any type: • Objects – Essentially Key Value Pairs – Values can be of any data type DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 21. All contents © MuleSoft Inc. Reading Data 21 • In Last slide we have seen methods of creating the data, now we will see how we can read data. • Dataweave selector will allow navigating any combination of Objects and Arrays to get the data we need. • Some of the selectors we will be looking into are: DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. Selector Representation Example Single Value Selector . Payload.name Index Selector [n] Payload[1] Range Selector [n to m] Payload[0..2] Multi Value Selector * Payload.*name Descendants Selector .. Payload..echo
- 22. All contents © MuleSoft Inc. Variables and Operators 22 • Variables – Way to store a value that can be used at later point of time – Makes code cleaner and readable – Performance Efficient as stored value can be reused. • Operators – Mathematical Operators • Addition (+), Subtraction (-), Multiplication (*), Division (/) – Equality and Relational Operators • < (Greater Than), > (Less Than), <=(Less Than or Equal to), >= (Greater Than or equal to), == (Equal To) – Logical Operators • Not, !, and, or DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 23. All contents © MuleSoft Inc. Flow Control 23 • Flow control is used when you want to execute certain parts of your code in some situations, while not executing others • In other words, it’s a way to add logic to your scripts. • If else.. – If/else expressions allow you to make decisions using logical operators and branch as a result. – if (<criteria_expression>) <return_if_true> else <return_if_false> • If else If: – Multiple conditions needed to be checked and alternate for case statement if (<criteria_expression1>) <return_if_true> else if (<criteria_expression2>) <return_if_true> else <return_if_no_other_match> DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 24. All contents © MuleSoft Inc. Flow Control 24 • Pattern matching is another method of flow control. • it does quite a bit more under the hood than the if/else expression does. • Like the if/else expression, pattern matching also returns a single value. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 25. All contents © MuleSoft Inc. Functions 25 • Functions are one of Data Weave's most important tools • Allow us to conveniently reuse functionality and create functionality on the fly. • The following are the different kinds of functions • Named functions • Lambdas • Passing functions to other functions • Calling 2-arity functions with infix notation • $, $$, $$$ syntax • Named Functions: – Created with fun keyword. – associates a set of functionality with a name – No return Keyword DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 26. All contents © MuleSoft Inc. Functions 26 • Dataweave allows us to use various ways to create functions • Functions without names are called as Lambdas. • Lambdas can be written as follows • For Lambda to execute, we surround it in parentheses and append () to the end. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 27. All contents © MuleSoft Inc. Working with Arrays 27 • We will look at the following aspects of the Arrays – Filter – Map – Distinct by – Group by – reduce DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 28. All contents © MuleSoft Inc. Filter 28 • filter(Array<T>, (item: T, index: Number) -> Boolean): Array<T> • Iterates over an array and applies an expression that returns matching values. • The expression must return true or false • If the expression returns true for a value or index in the array, the value gets captured in the output array. • If it returns false for a value or index in the array, that item gets filtered out of the output. • If there are no matches, the output array will be empty. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 29. All contents © MuleSoft Inc. Map 29 • map(Array<T>, (item: T, index: Number) -> R): Array<R> • map contains key value pairs • Example: %dw 2.0 output application/json --- ["jose", "pedro", "mateo"] map (value, index) -> { (index) : value} DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 30. All contents © MuleSoft Inc. Distinct By 30 • Iterates over an array and returns the unique elements in it. • Other versions act on an object and handle a null value • DataWeave uses the result of applying the provided lambda as the uniqueness criteria. %dw 2.0 output application/json --- [0, 1, 2, 3, 3, 2, 1, 4] distinctBy (value) -> { "unique" : value } DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 31. All contents © MuleSoft Inc. Group By 31 • Returns an object that groups items from an array based on specified criteria, such as an expression or matching selector. • This version of groupBy groups the elements of an array using the criteria function. Other versions act on objects and handle null values. • Example: %dw 2.0 var myArray = [ { "name": "Foo", "language": "Java" }, { "name": "Bar", "language": "Scala" }, { "name": "FooBar", "language": "Java" } ] output application/json --- myArray groupBy (item) -> item.language DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 32. All contents © MuleSoft Inc. Reduce 32 • Applies a reduction expression to the elements in an array. • or each element of the input array, in order, reduce applies the reduction lambda expression (function), then replaces the accumulator with the new result. • The lambda expression can use both the current input array element and the current accumulator value. • Example: %dw 2.0 output application/json --- [2, 3] reduce ($ + $$) DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 33. All contents © MuleSoft Inc. Working with Objects 33 • We will cover the following aspects of Dataweave with objects – Filter Object – Map Object – Pluck – Update DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 34. All contents © MuleSoft Inc. Working with Objects – Filter Object 34 • Iterates a list of key-value pairs in an object and applies an expression that returns only matching objects, filtering out the rest from the output. • The expression must return true or false. • If the expression returns true for a key, value, or index of an object, the object gets captured in the output. • If it returns false for any of them, the object gets filtered out of the output. • If there are no matches, the output array will be empty. filterObject({ (K)?: V }, (value: V, key: K, index: Number) -> Boolean): { (K)?: V } Example: %dw 2.0 output application/json --- {"a" : "apple", "b" : "banana"} filterObject ((value) -> value == "apple") DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 35. All contents © MuleSoft Inc. Working with Objects – Map Object 35 • Iterates over an object using a mapper that acts on keys, values, or indices of that object. • Example: %dw 2.0 output application/json --- {"a":"b","c":"d"} mapObject (value,key,index) -> { (index) : { (value):key} } Example: %dw 2.0 output application/xml --- { prices: payload.prices mapObject (value, key) -> { (key): (value + 5) as Number {format: "##.00"} } DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 36. All contents © MuleSoft Inc. Working with Objects – Pluck 36 • Useful for mapping an object into an array, pluck iterates over an object and returns an array of keys, values, or indices from the object. • It is an alternative to mapObject, which is similar but returns an object, instead of an array. • Example: %dw 2.0 output application/json --- {"a":"b","c":"d"} pluck (value,key,index) -> { (index) : { (value):key} } DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers. DataWeave also supports numbers with the Number type, covering both integer and floating-point numbers.
- 37. All contents © MuleSoft Inc. Trivia….! 37
- 38. All contents © MuleSoft Inc. Trivia 1 ..! 38 what is the daily limit for # of batches and records possible in a 24hr window a. 1000 batches and 100,000 b. 20000 batches and 200,000,000 c. 15000 batches and 150,000,000 d. None of the above
- 39. All contents © MuleSoft Inc. Trivia 2 ..! 39 What is the default batch size split in salesforce bulk api v2 ? a. 10000 b. 20000 c. 1000 d. None of the above
- 40. All contents © MuleSoft Inc. Trivia 3 ..! 40 What is the URI/path for getting unfinished results from a salesforce job ? a. /services/data/vXX.X/jobs/jobID/ingest/unprocessedrecords b. /services/data/vXX.X/jobs/ingest/jobID/unprocessedrecords c. /services/data/vXX.X/jobs/unprocessedrecords d. None of the above
- 41. All contents © MuleSoft Inc. Trivia 4 ..! 41 What DataWeave 2.0 type can be used as input to a map operation? a. Object b. String c. Array d. Map
- 42. All contents © MuleSoft Inc. Trivia 5 ..! 42 What does the zip operator do in DataWeave? a. Minifies the size of value using encoding. b. Merges elements of two objects into a single object. c. Merges elements of two lists (arrays) into a single list. d. None of these
- 43. All contents © MuleSoft Inc. Trivia 6 ..! 43 What does the minus operator do in DataWeave? A. Decrements the value by one. B. Removes items from a list. C. Increments the value by one. D. Removes characters from a string.
- 44. All contents © MuleSoft Inc. Trivia 7 ..! 44 How can you call a flow from Dataweave? A. Not allowed B. Include function C. Tag function D. Look Up function
- 45. All contents © MuleSoft Inc. Trivia 8 ..! 45 What is the value of the stepVar variable after the processing of records in a Batch Job? A. Null B. 0 C. - D. Last value from flow
- 46. Explore our new version MuleSoft Documentation
- 47. All contents © MuleSoft Inc. Find the answers you need, fast. 47 https://docs.mulesoft.com/
- 48. The best place to ask questions and help others. MuleSoft Help Center
- 49. All contents © MuleSoft Inc. 15,000+ members ready to help. 49 https://help.mulesoft.com/ • Check out the “MuleSoft Training” category for all training and certification- related questions
- 50. All contents © MuleSoft Inc. Join Charlotte MuleSoft Discussion Group. 50 https://help.mulesoft.com/s/ group/0F92T0000004odaSAA /charlotte-meetups • Check out the “Charlotte MuleSoft Discussion Group.” category for all your questions which you want to discuss within the Charlotte Group.
- 51. Networking time
- 52. All contents © MuleSoft Inc. Take a stand ! 52 • Nominate yourself for the next meetup speaker and suggest a topic as well.
- 53. All contents © MuleSoft Inc. What’s next 53 • Feedback: – Contact your organizer Subhash Patel or Aravind Ramadugu to suggest topics. – Contact MuleSoft at [email protected] for ways to improve the program.
- 54. See you soon…. Don’t forget to send topics to organizer
- 55. Thank you