DataFrames: The Extended Cut
19 likes8,471 views
Wes McKinney gave a talk at the 2015 Open Data Science Conference about data frames and the state of data frame interfaces across different languages and libraries. He discussed the challenges of collaboration between different data frame communities due to the tight coupling of user interfaces, data representations, and computation engines in current data frame implementations. McKinney predicted that over time these components would decouple and specialize, improving code sharing across languages.
![14
©
Cloudera,
Inc.
All
rights
reserved.
Data
types
{
‘name’: {
‘first’: ‘Wes’,
‘last’: ‘McKinney’
}
‘age’: 30
‘numbers’: [1, 2, 3, 4]
‘addresses’: [
{‘city’: ‘New York’,
‘state’: ‘NY’},
{‘city’: ‘San Francisco’,
‘state’: ‘CA’},
]
‘keys’: [(‘foo’, 27), (‘baz’, 35)]
}
struct<
name: struct<
first: string,
last: string
>,
age: int32,
numbers: list<int32>,
addresses: list<struct<
city: string,
state: string
>>,
keys: map<string, int>
>
Example
Table
Row
Table
schema](https://image.slidesharecdn.com/dataframesslides-150601035224-lva1-app6891/85/DataFrames-The-Extended-Cut-14-320.jpg)
![25
©
Cloudera,
Inc.
All
rights
reserved.
Spark
DataFrames
• R/pandas-‐inspired
API
for
tabular
data
manipulaQon
in
Scala,
Python,
etc.
• Logical
operaQon
graphs
rewrimen
internally
in
more
efficient
form
• Good
interop
with
Spark
SQL
• Some
interoperability
with
pandas
• ParQal
API
Decoupling!
• Low-‐level
internals
(DataFrame
~
RDD[Row])
are
…
not
the
best](https://image.slidesharecdn.com/dataframesslides-150601035224-lva1-app6891/85/DataFrames-The-Extended-Cut-25-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (16)
Ad
Similar to DataFrames: The Extended Cut (20)
Ad
More from Wes McKinney (16)
Recently uploaded (20)
DataFrames: The Extended Cut
- 1. 1 © Cloudera, Inc. All rights reserved. DataFrames: The Extended Cut Wes McKinney @wesmckinn Open Data Science Conference, 2015-‐05-‐31
- 2. 2 © Cloudera, Inc. All rights reserved. This talk • Some commentary / notes on all the data frame interfaces out there • Community / collaboraQon challenges • Opinions
- 3. 3 © Cloudera, Inc. All rights reserved. Disclaimer: This is a nuanced discussion
- 4. 4 © Cloudera, Inc. All rights reserved. Who am I? • Originator of pandas (2008 -‐ ) • Financial analyQcs in R / Python starQng 2007 • 2010-‐2012 • Hiatus from gainful employment • Make pandas ready for primeQme • Write "Python for Data Analysis” • 2013-‐2014: DataPad with Chang She & co • 2014 -‐ : Open source development @ Cloudera
- 5. 5 © Cloudera, Inc. All rights reserved. Data frame, in brief • A table-‐like data structure • An API / user interface for the table • SelecQng data • Math and relaQonal algebra (join, filter, etc.) • File / database IO • ad infinitum
- 6. 6 © Cloudera, Inc. All rights reserved. Some axes of comparison • Data structure internals (types, in-‐memory representaQon, etc.) • Basic table API • RelaQonal algebra support • Group-‐by / split-‐apply-‐combine API • Performance, memory use, evaluaQon semanQcs • Missing data • Data Qdying / ETL tools • IO uQliQes • Domain specific tools (e.g. Qme series) • …
- 7. 7 © Cloudera, Inc. All rights reserved. The Great Data Tool Decoupling™ • Thesis: over Qme, user interfaces, data storage, and execuQon engines will decouple and specialize • In fact, you should really want this to happen • Share systems among languages • Reduce fragmentaQon and “lock-‐in” • Shih developer focus to usability • PredicQon: we’ll be there by 2025; sooner if we all get our act together
- 8. 8 © Cloudera, Inc. All rights reserved. Crahing quality data tools • Quality / usefulness is usually forged by the fire of bamle • Real world use cases and social proof trump theory • Eat that dog food • When in doubt? Look at the test suite.
- 9. 9 © Cloudera, Inc. All rights reserved. Data frames, circa 2015 • Tight coupling amongst • User API • In-‐memory data representaQon • SerializaQon/deserializaQon tools • AnalyQcs / computaQon • Code evaluaQon semanQcs • Code sharing amongst languages fairly difficult, rarely happens in pracQce
- 10. 10 © Cloudera, Inc. All rights reserved. In-‐memory representaQon a major problem • Any algorithms wrimen against a data frame implementaQon assume a parQcular custom • Memory layout • Type system (+ missing data representaQon) • Memory management strategy • Due to organic growth of libraries / language ecosystems, there was never an effort to reach any design consensus • Downstream symptoms: benchmark-‐driven development
- 11. 11 © Cloudera, Inc. All rights reserved. Lies, damn lies, and benchmarks • Usual targets of benchmarking: • IO (CSV and database reading) • Joins • AggregaQon / group-‐by operaQons • What’s actually being tested • Quality of algorithm implementaQon • Data representaQon • Memory access pamerns / CPU cache efficiency
- 12. 12 © Cloudera, Inc. All rights reserved. Example: group-‐by-‐aggregate df.groupby(‘a’) .b.sum() SELECT a, sum(b) AS total FROM df GROUP BY 1 df %>% group_by(‘a’) %>% summarise(total=sum(b)) What’s actually being benchmarked • CPU/IO efficiency for for scanning a and b columns • Speed to push a values through a hash table (quality of hash table impl now an issue) • Time to sum b values given known categorizaQon (using hash table) • Speed of creaQng result data frame
- 13. 13 © Cloudera, Inc. All rights reserved. Data types • PrimiQve value types • Number (integer, floaQng point) • Boolean • String (UTF-‐8), Binary • Timestamp • Complex / nested types • Lists • Structs • Maps
- 14. 14 © Cloudera, Inc. All rights reserved. Data types { ‘name’: { ‘first’: ‘Wes’, ‘last’: ‘McKinney’ } ‘age’: 30 ‘numbers’: [1, 2, 3, 4] ‘addresses’: [ {‘city’: ‘New York’, ‘state’: ‘NY’}, {‘city’: ‘San Francisco’, ‘state’: ‘CA’}, ] ‘keys’: [(‘foo’, 27), (‘baz’, 35)] } struct< name: struct< first: string, last: string >, age: int32, numbers: list<int32>, addresses: list<struct< city: string, state: string >>, keys: map<string, int> > Example Table Row Table schema
- 15. 15 © Cloudera, Inc. All rights reserved. Table memory layout • Columnar • Faster for analyQcs, single-‐column scans • ProjecQons (column add/remove) cheap • Row appends harder • OperaQons across single rows are slower • Row-‐oriented • Slower single-‐column scans • ProjecQons expensive • Row appends easier • OperaQons across single rows are faster
- 16. 16 © Cloudera, Inc. All rights reserved. SerializaQon and protocols • Text-‐based formats • CSV/TSV, JSON • Binary formats • Avro • Thrih • Protocol buffers • Parquet (related: ColumnIO and RecordIO at Google) • ORCFile • HDF5 • Language specific: NumPy, bcolz, Rdata, etc.
- 17. 17 © Cloudera, Inc. All rights reserved. Lossy / text-‐based formats • Like CSV, JSON, etc. • Can be read “easily” by anyone • Expensive to read (parse) and write/generate • Expensive to infer correct schema if not known • SQll must validate parsed data, even if you know the schema • Not a high fidelity format • Schema must be known • Data can be easily lost-‐in-‐translaQon • CSV cannot easily handle nested schemas
- 18. 18 © Cloudera, Inc. All rights reserved. Binary data formats • Some, like Parquet and ORCFile, are columnar / analyQcs-‐oriented • Others (Avro, Thrih, Protobuf), designed for more general data transport and remote procedure calls (RPC) in distributed systems • Language-‐specific formats generally don’t have full-‐featured reader-‐writers outside the primary language
- 19. 19 © Cloudera, Inc. All rights reserved. Data representaQon across languages • R: list of named R arrays (each containing data of a primiQve R type) • Data frames have an implicit schema, but user does not interact with • Limited support for nested type values (JSON-‐like data) • pandas: complex, but fundamentally data stored in NumPy arrays • No explicit table schema; NumPy complexiQes largely hidden from user • Limited / no built-‐in support for nested type values • Missing / NULL values handled on a type-‐by-‐type basis (someQmes not at all)
- 20. 20 © Cloudera, Inc. All rights reserved. Missing data • R: uses special values to encode NA • Python pandas: special values, but does not work for all types • For arbitrary type data, best approach is to use a bit or byte to represent nullness/not-‐nullness
- 21. 21 © Cloudera, Inc. All rights reserved. I am acQvely working to address infrastructural issues that have prevented collaboraQon / code sharing in tabular data tools
- 22. 22 © Cloudera, Inc. All rights reserved. Some awesome R data frame stuff • “Hadley Stack” • dplyr, Qdyr • legacy: plyr, reshape2 • ggplot2 • data.table (data.frame + indices, fast algorithms) • xts : Qme series
- 23. 23 © Cloudera, Inc. All rights reserved. R data frames: rough edges • Copy-‐on-‐write semanQcs • API fragmentaQon / inconsistency • Use the “Hadley stack” for improved sanity • Factor / String dichotomy • stringsAsFactors=FALSE a blessing and curse • Somewhat limited type system
- 24. 24 © Cloudera, Inc. All rights reserved. dplyr • Composable table API • Good example of what the “decoupled” future might look like • New in-‐memory R/RCpp execuQon engine • SQL backends for large subset of API
- 25. 25 © Cloudera, Inc. All rights reserved. Spark DataFrames • R/pandas-‐inspired API for tabular data manipulaQon in Scala, Python, etc. • Logical operaQon graphs rewrimen internally in more efficient form • Good interop with Spark SQL • Some interoperability with pandas • ParQal API Decoupling! • Low-‐level internals (DataFrame ~ RDD[Row]) are … not the best
- 26. 26 © Cloudera, Inc. All rights reserved. pandas • Several key data structures, data frame among them • Considerably more complex internals than other data frame libraries • Some good things • Born of need • A “bameries included” approach • Hierarchical axis labeling: addresses some hard use cases at expense of semanQc complexity • Strong Qme series support
- 27. 27 © Cloudera, Inc. All rights reserved. pandas: rough edges • Axis labelling can get in the way for folks needing “just a table” • Ceded control of its type system / data rep’n from day 1 to NumPy • Inefficient string handling (uses NumPy object arrays) • Missing data handling less precise than other tools • No C API; very difficult for external tools to interact with pandas objects
- 28. 28 © Cloudera, Inc. All rights reserved. blaze • Open source Python project run by ConQnuum AnalyQcs • Bespoke type system (“datashape”) supporQng many kinds of nested schemas • Data expression API with many execuQon backends (SQL, pandas, Spark, etc.) • Does not have a “naQve” backend (was to be the—now aborted?—libdynd project) • Another good example of the “decoupled” future
- 29. 29 © Cloudera, Inc. All rights reserved. libdynd • Next-‐gen NumPy-‐in-‐C++ project started by ConQnuum in 2011 to be a naQve backend for Blaze, appears not to be acQvely pursued • Implements the new datashape protocol • Standalone C++11/14 library, with deep Python binding • Lots of interesQng internals / implementaQon ideas
- 30. 30 © Cloudera, Inc. All rights reserved. bcolz: the new carray / ctable • Compressed in-‐memory / on disk columnar table structure • Outgrowth of • carray (compressed-‐in-‐memory NumPy array) • blosc (mulQthreaded shuffling compression library)
- 31. 31 © Cloudera, Inc. All rights reserved. Julia: DataFrames.jl • Started by Harlan Harris & co • Part of broader JuliaStats iniQaQve • More R-‐like than pandas-‐like • Very acQve: > 50 contributors! • SQll comparaQvely early • Less comprehensive API • More limited IO capabiliQes
- 32. 32 © Cloudera, Inc. All rights reserved. Other data frames • Saddle (Scala) • Dev’d by Adam Klein (ex-‐AQR) at Novus Partners (fintech startup) • Designed and used for financial use cases • Deedle (F# / .NET) • Dev’d by AK’s colleagues at BlueMountain (hedge fund) • GraphLab / Dato • Really good C++ data frame with Python interface • Dual-‐licensed: AGPL + Commercial • That’s not all! Haskell, Go, etc…
- 33. 33 © Cloudera, Inc. All rights reserved. We’re not done yet • The future is JSON-‐like • Support for nested types / semi-‐structured data is sQll weak • Wanted: Apache-‐licensed, community standard C/C++ data frame data structure and analyQcs toolkit that we all use (R, Python, Julia) • Bring on the Great Decoupling
- 34. 34 © Cloudera, Inc. All rights reserved. Thank you @wesmckinn