U-SQL Intro (SQLBits 2016)
Download as PPTX, PDF0 likes471 views
The document discusses Azure Data Lake and U-SQL, highlighting its ability to unify structured and unstructured data processing to enhance productivity and agility. It outlines U-SQL's declarative syntax, extensibility with C#/.NET, and features like user-defined functions and parallel processing. Additionally, it emphasizes the ease of use and scalability, with support for complex data manipulation and performance optimization.
![DECLARE Constants/U-SQL Types
DECLARE @text1 string = "Hello World";
DECLARE @text2 string = @"Hello World";
DECLARE @text3 char = 'a';
DECLARE @text4 string = "BEGIN" + @text1 + "END";
DECLARE @text5 string = string.Format("BEGIN{0}END", @text1);
DECLARE @numeric1 sbyte = 0;
DECLARE @numeric2 short = 1;
DECLARE @numeric3 int = 2;
DECLARE @numeric4 long = 3L;
DECLARE @numeric5 float = 4.0f;
DECLARE @numeric6 double = 5.0;
DECLARE @d1 DateTime = System.DateTime.Parse("1979/03/31");
DECLARE @d2 DateTime = DateTime.Now;
DECLARE @misc1 bool = true;
DECLARE @misc2 Guid = System.Guid.Parse("BEF7A4E8-F583-4804-9711-7E608215EBA6");
DECLARE @misc4 byte [] = new byte[] { 0, 1, 2, 3, 4};
DECLARE @map SqlMap<string,string> =
new SqlMap<string,string> {{"key","value"}, {"key","value"});
DECLARE @arr SqlArray<string> = new SqlArray<string> {"val1", "val2"});
DECLARE @nullableint int? = null;
text
numeric
Date/time
Other](https://image.slidesharecdn.com/2-sqlbits-usqlintro-160505123813/85/U-SQL-Intro-SQLBits-2016-11-320.jpg)
U-SQL Intro (SQLBits 2016)
- 1. SQLBits 2016 Azure Data Lake & U-SQL Michael Rys, @MikeDoesBigData http://www.azure.com/datalake {mrys, usql}@microsoft.com
- 4. • • •
- 5. hard to work with anything other than structured data difficult to extend with custom code
- 6. User often has to care about scale and performance SQL is 2nd class within string Often no code reuse/ sharing across queries
- 7. Get benefits of both! Makes it easy for you by unifying: • Unstructured and structured data processing • Declarative SQL and custom imperative Code • Local and remote Queries • Increase productivity and agility from Day 1 and at Day 100 for YOU!
- 9. Extend U-SQL with C#/.NET Built-in operators, function, aggregates C# expressions (in SELECT expressions) User-defined aggregates (UDAGGs) User-defined functions (UDFs) User-defined operators (UDOs)
- 11. DECLARE Constants/U-SQL Types DECLARE @text1 string = "Hello World"; DECLARE @text2 string = @"Hello World"; DECLARE @text3 char = 'a'; DECLARE @text4 string = "BEGIN" + @text1 + "END"; DECLARE @text5 string = string.Format("BEGIN{0}END", @text1); DECLARE @numeric1 sbyte = 0; DECLARE @numeric2 short = 1; DECLARE @numeric3 int = 2; DECLARE @numeric4 long = 3L; DECLARE @numeric5 float = 4.0f; DECLARE @numeric6 double = 5.0; DECLARE @d1 DateTime = System.DateTime.Parse("1979/03/31"); DECLARE @d2 DateTime = DateTime.Now; DECLARE @misc1 bool = true; DECLARE @misc2 Guid = System.Guid.Parse("BEF7A4E8-F583-4804-9711-7E608215EBA6"); DECLARE @misc4 byte [] = new byte[] { 0, 1, 2, 3, 4}; DECLARE @map SqlMap<string,string> = new SqlMap<string,string> {{"key","value"}, {"key","value"}); DECLARE @arr SqlArray<string> = new SqlArray<string> {"val1", "val2"}); DECLARE @nullableint int? = null; text numeric Date/time Other
- 12. U-SQL Language Philosophy Declarative Query and Transformation Language: • Uses SQL’s SELECT FROM WHERE with GROUP BY/Aggregation, Joins, SQL Analytics functions • Optimizable, Scalable Expression-flow programming style: • Easy to use functional lambda composition • Composable, globally optimizable Operates on Unstructured & Structured Data • Schema on read over files • Relational metadata objects (e.g. database, table) Extensible from ground up: • Type system is based on C# • Expression language IS C# • User-defined functions (U-SQL and C#) • User-defined Aggregators (C#) • User-defined Operators (UDO) (C#) U-SQL provides the Parallelization and Scale-out Framework for Usercode • EXTRACTOR, OUTPUTTER, PROCESSOR, REDUCER, COMBINER, APPLIER Federated query across distributed data sources REFERENCE MyDB.MyAssembly; CREATE TABLE T( cid int, first_order DateTime , last_order DateTime, order_count int , order_amount float ); @o = EXTRACT oid int, cid int, odate DateTime, amount float FROM "/input/orders.txt" USING Extractors.Csv(); @c = EXTRACT cid int, name string, city string FROM "/input/customers.txt" USING Extractors.Csv(); @j = SELECT c.cid, MIN(o.odate) AS firstorder , MAX(o.date) AS lastorder, COUNT(o.oid) AS ordercnt , AGG<MyAgg.MySum>(c.amount) AS totalamount FROM @c AS c LEFT OUTER JOIN @o AS o ON c.cid == o.cid WHERE c.city.StartsWith("New") && MyNamespace.MyFunction(o.odate) > 10 GROUP BY c.cid; OUTPUT @j TO "/output/result.txt" USING new MyData.Write(); INSERT INTO T SELECT * FROM @j;
- 13. 2 Automatic "in-lining" optimized out-of- the-box Per job parallelization visibility into execution Heatmap to identify bottlenecks
Editor's Notes
- #5: Add velocity?
- #6: Hard to operate on unstructured data: Even Hive requires meta data to be created to operate on unstructured data. Adding Custom Java functions, aggregators and SerDes is involving a lot of steps and often access to server’s head node and differs based on type of operation. Requires many tools and steps. Some examples: Hive UDAgg Code and compile .java into .jar Extend AbstractGenericUDAFResolver class: Does type checking, argument checking and overloading Extend GenericUDAFEvaluator class: implements logic in 8 methods. - Deploy: Deploy jar into class path on server Edit FunctionRegistry.java to register as built-in Update the content of show functions with ant Hive UDF (as of v0.13) Code Load JAR into head node or at URI CREATE FUNCTION USING JAR to register and load jar into classpath for every function (instead of registering jar and just use the functions)
- #7: Spark supports Custom “inputters and outputters” for defining custom RDDs No UDAGGs Simple integration of UDFs but only for duration of program. No reuse/sharing. Cloud dataflow? Requires has to care about scale and perf Spark UDAgg Is not yet supported ( SPARK-3947) Spark UDF Write inline function def westernState(state: String) = Seq("CA", "OR", "WA", "AK").contains(state) for SQL usage need to register the table customerTable.registerTempTable("customerTable") Register each UDF sqlContext.udf.register("westernState", westernState _) Call it val westernStates = sqlContext.sql("SELECT * FROM customerTable WHERE westernState(state)")
- #8: Offers Auto-scaling and performance Operates on unstructured data without tables needed Easy to extend declaratively with custom code: consistent model for UDO, UDF and UDAgg. Easy to query remote sources even without external tables U-SQL UDAgg Code and compile .cs file: Implement IAggregate’s 3 methods :Init(), Accumulate(), Terminate() C# takes case of type checking, generics etc. Deploy: Tooling: one click registration in user db of assembly By Hand: Copy file to ADL CREATE ASSEMBLY to register assembly Use via AGG<MyNamespace.MyAggregate<T>>(a) U-SQL UDF Code in C#, register assembly once, call by C# name.
- #9: Remove SCOPE for external customers?
- #10: Extensions require .NET assemblies to be registered with a database
- #13: Use for language experts