Building a modern Application with DataFrames

Editor's Notes

• #13: This saturated both disk and network layers
• #15: Old Spark API (T&A) is based on Java/Python objects - this makes it hard for the engine to store compactly (java objects in memory have a lot of extra space for what classes, pointers to various things, etc) - cannot understand semantics of user functions - so if you run a map function over just one field of the data, it still has to read the entire object into memory. Spark doesn't know you only cared about one field. 
• #17: DataFrames were inspired by previous distributed data frame efforts, including Adatao’s DDF and Ayasdi’s BigDF. However, the main difference from these projects is that DataFrames go through the Catalyst optimizer, enabling optimized execution similar to that of Spark SQL queries.
• #18: a DataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood.  I’d say that DataFrame is a result of transformation of any other RDD. Your input RDD might contains strings and numbers. But as a result of transformation you end up with RDD that contains GenericRowWithSchema, which is what DataFrame actually is. So, I’d say that DataFrame is just sort of wrapper around simple RDD, which provides some additional and pretty useful stuff.
• #21: To compute an average. I have a dataset that is a list of names and ages. Want to figure out the average age for a given name. So, age distribution for a name…
• #26: Head is non-deterministic, could change between jobs. Just the first partition that materialized returns results.
• #27: Head is non-deterministic, could change between jobs. Just the first partition that materialized returns results.
• #29: SparkSQL is the only project (1.4+) can read from multiple version of Hive. Spark 1.5 can read from 0.12 – 1.2 A lot of the hive functionality is useful even if you don’t have a Hive installation! Spark will automatically create a local copy of the Hive metastore so use can use Window functions, Hive UDFS, create persistent tables, To use a HiveContext, you do not need to have an existing Hive setup, and all of the data sources available to a SQLContext are still available. HiveContext is only packaged separately to avoid including all of Hive’s dependencies in the default Spark build.  The specific variant of SQL that is used to parse queries can also be selected using the spark.sql.dialect option. This parameter can be changed using either the setConf method on a SQLContext or by using a SET key=valuecommand in SQL. For a SQLContext, the only dialect available is “sql” which uses a simple SQL parser provided by Spark SQL. In a HiveContext, the default is “hiveql”, though “sql” is also available. 
• #30: The following example shows how to construct DataFrames in Python. A similar API is available in Scala and Java.
• #31: Once built, DataFrames provide a domain-specific language for distributed data manipulation.  Here is an example of using DataFrames to manipulate the demographic data of a large population of users:
• #32: You can also incorporate SQL while working with DataFrames, using Spark SQL. This example counts the number of users in the young DataFrame.
• #34:  But its not the same as if you called .cache() on an RDD[Row], since we reencode the data into bytebuffers before calling caching so that there is less pressure on the GC.
• #38: https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrame the full RDD API cannot be released on Dataframes. Per Michael, absolute freedom for users restricts the types of optimizations that we can do.
• #43: Finally, a Data Source for reading from JDBC has been added as built-in source for Spark SQL.  Using this library, Spark SQL can extract data from any existing relational databases that supports JDBC.  Examples include mysql, postgres, H2, and more.  Reading data from one of these systems is as simple as creating a virtual table that points to the external table.  Data from this table can then be easily read in and joined with any of the other sources that Spark SQL supports. This functionality is a great improvement over Spark’s earlier support for JDBC (i.e.,JdbcRDD).  Unlike the pure RDD implementation, this new DataSource supports automatically pushing down predicates, converts the data into a DataFrame that can be easily joined, and is accessible from Python, Java, and SQL in addition to Scala.
• #45: Twitter and Cloudera merged efforts in 2012 to develop a columnar format Parquet is a column based storage format. It gets its name from the patterns in parquet flooring. Optimized use case for parquet is when you only need a subset of the total columns. Avro is better if you typically scan/read all of the fields in a row in each query. Typically, one of the most expensive parts of reading and writing data is (de)serialization. Parquet supports predicate push-down and schema projection to target specific columns in your data for filtering and reading — keeping the cost of deserialization to a minimum. Parquet compresses better because columns have a fixed data type (like string, integer, Boolean, etc).  it is easier to apply any encoding schemes on columnar data which could even be column specific such as delta encoding for integers and prefix/dictionary encoding for strings. Also, due to the homogeneity in data, there is a lot more redundancy and duplicates in the values in a given column. This allows better compression in comparison to data stored in row format. The format is explicitly designed to separate the metadata from the data. This allows splitting columns into multiple files, as well as having a single metadata file reference multiple parquet files. Ideal row group size: 512 MB – 1 GB. Since an entire row group might need to be read, we want it to completely fit on one HDFS block. Therefore, HDFS block sizes should also be set to be larger. An optimized read setup would be: 1GB row groups, 1GB HDFS block size, 1 HDFS block per HDFS file. Larger row groups allow for larger column chunks which makes it possible to do larger sequential IO. Larger groups also require more buffering in the write path (or a two pass write).  Data page size: 8 KB recommended. Data pages should be considered indivisible so smaller data pages allow for more fine grained reading (e.g. single row lookup). Larger page sizes incur less space overhead (less page headers) and potentially less parsing overhead (processing headers).  https://parquet.apache.org/documentation/latest/
• #46: Row group: A logical horizontal partitioning of the data into rows. There is no physical structure that is guaranteed for a row group. A row group consists of a column chunk for each column in the dataset. Column chunk: A chunk of the data for a particular column. These live in a particular row group and is guaranteed to be contiguous in the file. Page: Column chunks are divided up into pages. A page is conceptually an indivisible unit (in terms of compression and encoding). There can be multiple page types which is interleaved in a column chunk. Hierarchically, a file consists of one or more row groups. A row group contains exactly one column chunk per column. Column chunks contain one or more pages. Metadata is written after the data to allow for single pass writing. Readers are expected to first read the file metadata to find all the column chunks they are interested in. The columns chunks should then be read sequentially. There are three types of metadata: file metadata, column (chunk) metadata and page header metadata. All thrift structures are serialized using the TCompactProtocol. The format is explicitly designed to separate the metadata from the data. This allows splitting columns into multiple files, as well as having a single metadata file reference multiple parquet files. Ideal row group size: 512 MB – 1 GB. Since an entire row group might need to be read, we want it to completely fit on one HDFS block. Therefore, HDFS block sizes should also be set to be larger. An optimized read setup would be: 1GB row groups, 1GB HDFS block size, 1 HDFS block per HDFS file. Larger row groups allow for larger column chunks which makes it possible to do larger sequential IO. Larger groups also require more buffering in the write path (or a two pass write).  Data page size: 8 KB recommended. Data pages should be considered indivisible so smaller data pages allow for more fine grained reading (e.g. single row lookup). Larger page sizes incur less space overhead (less page headers) and potentially less parsing overhead (processing headers).  https://parquet.apache.org/documentation/latest/
• #47: First, organizations that store lots of data in parquet often find themselves evolving the schema over time by adding or removing columns.  With this release we add a new feature that will scan the metadata for all files, merging the schemas to come up with a unified representation of the data.  This functionality allows developers to read data where the schema has changed overtime, without the need to perform expensive manual conversions. - In Spark 1.4, we plan to provide an interface that will allow other formats, such as ORC, JSON and CSV, to take advantage of this partitioning functionality.
• #50: On the builder you can specific methods… Like do you want to overwrite data already there? Load or save or saveAsTable are actions.
• #53: Note that by default in Spark SQL, there is a parameter called spark.sql.shuffle.partititions, which sets the # of partitions in a Dataframe after a shuffle (in case the user didn't manually specify it). Currently, Spark does not do any automatic determination of partitions, it just uses the # in that parameter. Doing more automclasses for Databrick stuff is on our roadmap though. You can change this parameter using: sqlContex.setConf(key, value). 1.6 = adaptive shuffle, look at output of map side, then pick # of reducers. Matie and Yin's hack day project. 
• #58: Case classes are used when creating classes that primarily hold data. When your class is basically a data-holder, case classes simplify your code and perform common work. With case classes, unlike regular classes, we don’t have to use the new keyword when creating an object. The Scala interface for Spark SQL supports automatically converting an RDD containing case classes to a DataFrame. The case class defines the schema of the table. The names of the arguments to the case class are read using reflection (seen in green) and become the names of the columns. Case classes can also be nested or contain complex types such as Sequences or Arrays. This RDD can be implicitly converted to a DataFrame and then be registered as a table. Note: Case classes in Scala 2.10 can support only up to 22 fields. To work around this limit, // you can use custom classes that implement the Product interface. - - - - What is a case class (vs a normal class)? Original purpose was used for matching, but it’s used for more now Scala’s version of a java bean (java has classes primary for data (gettings and settings) and there’s classes mostly for operations Case classes are mostly for data Scala can do reflection to establish/infer the schema of the df (seen in green). You’d want to be more robust about parsing CSV in real life. peopleRDD.toDF uses (a) Scala implicits and (b) the type of the RDD (RDD[Person]) to infer the schema Mention that a case class, in Scala, is basically a Scala bean: A container for data, augmented with useful things by the Scala compiler.
• #61: Catalyst is a powerful new optimization framework. The Catalyst framework allows the developers behind Spark SQL to rapidly add new optimizations, enabling us to build a faster system more quickly.  Unlike the eagerly evaluated data frames in R and Python, DataFrames in Spark have their execution automatically optimized by a query optimizer called Catalyst. Before any computation on a DataFrame starts, the Catalyst optimizer compiles the operations that were used to build the DataFrame into a physical plan for execution.
• #62: Unlike the eagerly evaluated data frames in R and Python, DataFrames in Spark have their execution automatically optimized by a query optimizer called Catalyst. Before any computation on a DataFrame starts, the Catalyst optimizer compiles the operations that were used to build the DataFrame into a physical plan for execution.
• #63: At a high level, there are two kinds of optimizations. First, Catalyst applies logical optimizations such as predicate pushdown. The optimizer can push filter predicates down into the data source, enabling the physical execution to skip irrelevant data. In the case of Parquet files, entire blocks can be skipped and comparisons on strings can be turned into cheaper integer comparisons via dictionary encoding. In the case of relational databases, predicates are pushed down into the external databases to reduce the amount of data traffic. Second, Catalyst compiles operations into physical plans for execution and generatesJVM bytecode for those plans that is often more optimized than hand-written code. For example, it can choose intelligently between broadcast joins and shuffle joins to reduce network traffic. It can also perform lower level optimizations such as eliminating expensive object allocations and reducing virtual function calls. As a result, we expect performance improvements for existing Spark programs when they migrate to DataFrames. Since the optimizer generates JVM bytecode for execution, Python users will experience the same high performance as Scala and Java users.
• #64: Since the optimizer generates JVM bytecode for execution, Python users will experience the same high performance as Scala and Java users. The above chart compares the runtime performance of running group-by-aggregation on 10 million integer pairs on a single machine (source code). Since both Scala and Python DataFrame operations are compiled into JVM bytecode for execution, there is little difference between the two languages, and both outperform the vanilla Python RDD variant by a factor of 5 and Scala RDD variant by a factor of 2.
• #66: At its core, Catalyst contains a general library for representing trees and applying rules to manipulate them.  A tree is just a Scala object. - - These classes can be used to build up trees; for example, the tree for the expression x+(1+2), would be represented in Scala code as follows: (See Scala code and Diagram)
• #67: Pattern matching is a feature of many functional languages that allows extracting values from potentially nested structures of algebraic data types. - The case keyword here is Scala’s standard pattern matching syntax, and can be used to match on the type of an object as well as give names to extracted values (c1 and c2 here). - The pattern matching expression that is passed to transform is a partial function, meaning that it only needs to match to a subset of all possible input trees.  This ability means that rules only need to reason about the trees where a given optimization applies and not those that do not match. Thus, rules do not need to be modified as new types of operators are added to the system.
• #68: Rules (and Scala pattern matching in general) can match… -  In the example above, repeated application would constant-fold larger trees, such as (x+0)+(3+3)
• #69: Examples below show why you may want to run a rule multiple times Running rules to fixed point means that each rule can be simple and self-contained, and yet still eventually have larger global effects on a tree.  - In our experience, functional transformations on immutable trees make the whole optimizer very easy to reason about and debug. They also enable parallelization in the optimizer, although we do not yet exploit this.
• #70:  In the physical planning phase, Catalyst may generate multiple plans and compare them based on cost. All other phases are purely rule-based. Each phase uses different types of tree nodes; Catalyst includes libraries of nodes for expressions, data types, and logical and physical operators. - Does Catalyst currently have the capability to generate multiple physical plans? You had mentioned at TtT last week that costing is done eagerly to prune branches that are not allowed (aka greedy algorithm).  - Greedy Algorithm works by making the decision that seems most promising at any moment; it never reconsiders this decision, whatever situation may arise later. A greedy algorithm is an algorithm that follows the problem solving heuristic of making the locally optimal choice at each stage with the hope of finding a global optimum.
• #71: Spark SQL begins with a relation to be computed, either from an abstract syntax tree (AST) returned by a SQL parser, or from a DataFrame object constructed using the API. A syntax tree is a tree representation of the structure of the source code. Each node of the tree denotes a construct occurring in the source code. The syntax is "abstract" in not representing every detail appearing in the real syntax. For instance, grouping parentheses are implicit in the tree structure, and a syntactic construct like an if-condition-then expression may be denoted by means of a single node with three branches. - An abstract syntax tree for the following code for the Euclidean algorithm:while b ≠ 0if a > ba := a − belseb := b − areturn a - -
• #72: Spark SQL begins with a relation to be computed, either from an abstract syntax tree (AST) returned by a SQL parser, or from a DataFrame object constructed using the API. A syntax tree is a tree representation of the structure of the source code. Each node of the tree denotes a construct occurring in the source code. The syntax is "abstract" in not representing every detail appearing in the real syntax. For instance, grouping parentheses are implicit in the tree structure, and a syntactic construct like an if-condition-then expression may be denoted by means of a single node with three branches. - An abstract syntax tree for the following code for the Euclidean algorithm:while b ≠ 0if a > ba := a − belseb := b − areturn a - -
• #73: A trivial [[Analyzer]] with an [[EmptyCatalog]] and [[EmptyFunctionRegistry]]. Used for testing * when all relations are already filled in and the analyser needs only to resolve attribute * references. * Provides a logical query plan analyzer, which translates [[UnresolvedAttribute]]s and * [[UnresolvedRelation]]s into fully typed objects using information in a schema [[Catalog]] and * a [[FunctionRegistry]].
• #74: Note, this is not cost based. Cost-based optimization is performed by generating multiple plans using rules, and then computing their costs.
• #75: A trivial [[Analyzer]] with an [[EmptyCatalog]] and [[EmptyFunctionRegistry]]. Used for testing * when all relations are already filled in and the analyser needs only to resolve attribute * references. * Provides a logical query plan analyzer, which translates [[UnresolvedAttribute]]s and * [[UnresolvedRelation]]s into fully typed objects using information in a schema [[Catalog]] and * a [[FunctionRegistry]].
• #76: The framework supports broader use of cost-based optimization, however, as costs can be estimated recursively for a whole tree using a rule. We thus intend to implement richer cost-based optimization in the future.
• #77: A trivial [[Analyzer]] with an [[EmptyCatalog]] and [[EmptyFunctionRegistry]]. Used for testing * when all relations are already filled in and the analyser needs only to resolve attribute * references. * Provides a logical query plan analyzer, which translates [[UnresolvedAttribute]]s and * [[UnresolvedRelation]]s into fully typed objects using information in a schema [[Catalog]] and * a [[FunctionRegistry]].
• #78: As a simple example, consider the Add, Attribute and Literal tree nodes introduced in Section 4.2, which allowed us to write expressions such as (x+y)+1. Without code generation, such expressions would have to be interpreted for each row of data, by walking down a tree of Add, Attribute and Literal nodes. This introduces large amounts of branches and virtual function calls that slow down execution. With code generation, we can write a function to translate a specific expression tree to a Scala AST as follows: The strings beginning with q are quasiquotes, meaning that although they look like strings, they are parsed by the Scala compiler at compile time and represent ASTs for the code within. Quasiquotes can have variables or other ASTs spliced into them, indicated using $ notation. For example, Literal(1) would become the Scala AST for 1, whileAttribute("x") becomes row.get("x"). In the end, a tree like Add(Literal(1), Attribute("x")) becomes an AST for a Scala expression like 1+row.get("x").
• #79: A trivial [[Analyzer]] with an [[EmptyCatalog]] and [[EmptyFunctionRegistry]]. Used for testing * when all relations are already filled in and the analyser needs only to resolve attribute * references. * Provides a logical query plan analyzer, which translates [[UnresolvedAttribute]]s and * [[UnresolvedRelation]]s into fully typed objects using information in a schema [[Catalog]] and * a [[FunctionRegistry]].
• #80: Sometimes you want to call complex functions to do additional work inside of the SQL queries. UDFs can be inlined in the DataFrame code UDF zipToCity just invokes a lamda function that takes a zipCode and does some custom logic to figure out which city the zip code is located in. I have a function called add demographics, which takes a data frame w/ a user ID and will automatically compute a bunch of demographic information. So, we do a join based on UserID and then adds a new column with the .withColumn… the UDF results will add a new column. This def returns a new DataFrame
• #81: All of this is lazy, so SparkSQL can do optimizations much later. For this type of machine learning I’m doing, I may only need the ts column from San Francisco. Note that add_demographics does have extra functionality to filter down to just SF and ts.
• #82: So maybe add_demographics was by my co-worker and I just want to use it. So we construct a logical query plan. Since this planning is happening at the logical level, optimizations can even occur across function calls, as shown in the example below. In this example, Spark SQL is able to push the filtering of users by their location through the join, greatly reducing its cost to execute.  This optimization is possible even though the original author of the add_demographics function did not provide a parameter for specifying how to filter users! - Ideally we want to filter the users ahead of time based on the extra predicates, and only do the join on the relevent users.
• #83: Even cooler, if I want to optimize this later on… So, here I changed to a partitioned hive table (users) and also used parquet instead of JSON (events). Now with Parquet, SparkSQL notices there are new optimizations that it can now do
• #84: Idea of Project Tungsten is to reimagine the execution engine for SparkSQL As a user, when you move to 1.5 you will see significant robustness and speed improvements
• #85: Before you had to resort to Hive UDFs or drop into SQL… But now there are 100+ native functions added that at runtime Java bytecode will be constructed to evaluate whatever you need.
• #88: Pass a physical plan generated by Catalyst into Streaming