Simplifying training deep and serving learning models with big data in python using tensorflow - PyData Berlin

Simplifying Training &
Serving Deep Learning
with Big Data in Python
w/ Tensorflow, Apache Beam, and Apache Flink
+ Possible Spark bonus
@holdenkarau

Holden:
● My name is Holden Karau
● Prefered pronouns are she/her
● Developer Advocate at Google
● Apache Spark PMC, Beam contributor
● previously IBM, Alpine, Databricks, Google, Foursquare & Amazon
● co-author of Learning Spark & High Performance Spark
● Twitter: @holdenkarau
● Slide share http://www.slideshare.net/hkarau
● Code review livestreams: https://www.twitch.tv/holdenkarau /
https://www.youtube.com/user/holdenkarau
● Spark Talk Videos http://bit.ly/holdenSparkVideos

Who I think you wonderful humans are?
● Nice enough people
● Don’t mind pictures of cats
● Probably pretty familiar with Spark
● Maybe somewhat familiar with Beam?
Lori Erickson

What we did get to:
● TensorFlowOnSpark w/basic Apache Arrow
● Python & Go on Beam on Flink prototype (in random
github branch, not finished)
○ Everyone loves wordcount right? right?....
● New Beam architecture allowing for better portability &
handling dependencies (like Tensorflow)
● TF Transform demo on Apache Flink via Apache Beam
● DO NOT non-JVM BEAM on Flink IN PRODUCTION
Vladimir Pustovit

DO NOT USE THIS IN PRODUCTION TODAY
● I’m serious, I don’t want to die or cause the next
financial meltdown with software I’m a part of
● By Today I mean June 8 2018, but it’s probably going to
not be great for at least a “little while”
Vladimir Pustovit
PROTambako The Jaguar

What will be covered?
Most likely:
● Where we are today in non-JVM support in Beam
● And why this matters for Tensorflow
● What the rest of Big Data ecosystem looks like going outside the JVM
● A partial TF Transform demo + TFMA links + possible system crash
If there is time (e.g. demo runs quickly, wifi works, several other happy unlikely
things):
● TensorFlowOnSpark
● Apache Arrow - How this changes “everything”*

So why do I need to power DL w/Big Data?
● Deep learning is most effective with large sample sets for training
● As much as some may say that no feature prep is required even if you’re
looking at mnist.csv you probably have _some_ feature prep
● Since we need big data for training we need to to do our feature prep it
● Even if your just trying to raise some VC money it's going to go a lot better if
you add some keywords about a large proprietary dataset

TensorFlow isn’t enough on its own
● Enter TFX & friends like Kubeflow
○ Current related TFX OSS components: TF.Transform TF.Serving (with
more coming)
● Alternatives: piles of custom code re-created at serving time.
○ Yay job security?
PROJennifer C.

How do I do feature prep? (old skool)
● Write custom preparation jobs in your favourite big data tool (mine is Apache
Spark, but maybe yours is something else)
● Run it, train on the prepared data
● Rewrite your feature prep code to run at serving time
○ Error prone and sad

Enter: TF.Transform
● For pre-processing of your data
○ e.g. where you spend 90% of your dev time anyways
● Integrates into serving time :D
● OSS
● Runs on top of Apache Beam, but current release not yet outside of GCP
○ On master this can run on Flink, but probably has bugs currently.
○ Please don’t use this in production today unless your on
GCP/Dataflow
PROKathryn Yengel

Defining a Transform processing function
def preprocessing_fn(inputs):
x = inputs['x']
y = inputs['y']
s = inputs['s']
x_centered = x - tft.mean(x)
y_normalized = tft.scale_to_0_1(y)
s_int = tft.string_to_int(s)
return { 'x_centered': x_centered,
'y_normalized': y_normalized, 's_int': s_int}

mean stddev
normalize
multiply
quantiles
bucketize
Analyzers
Reduce (full pass)
Implemented as a distributed
data pipeline
Transforms
Instance-to-instance (don’t
change batch dimension)
Pure TensorFlow

Analyze
normalize
multiply
bucketize
constant
tensors
data
mean stddev
normalize
multiply
quantiles
bucketize

Scale to ... Bag of Words / N-Grams
Bucketization Feature Crosses
tft.ngrams
tft.string_to_int
tf.string_split
tft.scale_to_z_score
tft.apply_buckets
tft.quantiles
tft.string_to_int
tf.string_join
...
Some common use-cases...

BEAM Beyond the JVM: Current release
● Non JVM BEAM doesn’t work outside of Google’s environment yet
● tl;dr : uses grpc / protobuf
○ Similar to the common design but with more efficient representations (often)
● But exciting new plans to unify the runners and ease the support of different
languages (called SDKS)
○ See https://beam.apache.org/contribute/portability/
● If this is exciting, you can come join me on making BEAM work in Python3
○ Yes we still don’t have that :(
○ But we're getting closer & you can come join us on BEAM-2874 :D
Emma

BEAM Beyond the JVM: Master + Experiments
● Common interface for setting up jobs
● Portability framework allows SDK harnesses in arbitrary to be kicked off
● Runners ship in their own docker containers (goodbye dependency hell, hello
container hell)
○ Also for now rolling containers leaves something to be desired (e.g. edit docker file by hand)
● Hacked up Python SDK to sort of talk to the new interface
● Go SDK talks to the new interface, still missing some features
● Need permissions? Run on GKE or plumb through permissions file :(
Nick

BEAM Beyond the JVM: Master w/ experiments
*ish
*ish
*ish
Nick
portability
*ish

So what does that look like?
Driver
Worker 1
Docker
grpc
Worker K
Docker
grpc

BEAM Beyond the JVM: The “future”
E.g. not now
*ish
*ish
*ish
Nick
portability
*ish
*ish

So how TF does this relate to TF?
● Tensorflow is in Python (kind of)
● Once we finish the Python SDK on Beam on Flink adventure you can use all
sorts of cool libraries (like TFT/TFX) to do your tensorflow work
○ You can use them today too if your use case is on Dataflow
○ If you don’t mind bugs you can experiment with them on Flink too
● You will be able manage your dependencies
● You will be able to (in theory) re-use dataprep code at serving time
○ 80% less copy n’ paste code with slight mistakes that get out of date!**
● No that doesn’t work today
● Or tomorrow
● But… eventually
○ Standard OSS excuse “patches welcome” (sort of if you can find the branch :p)
**Not a guarantee, see your vendor for details.

Ooor from the chicago taxi data...
for key in taxi.DENSE_FLOAT_FEATURE_KEYS:
# Preserve this feature as a dense float, setting nan's to
the mean.
outputs[key] = transform.scale_to_z_score(inputs[key])
for key in taxi.VOCAB_FEATURE_KEYS:
# Build a vocabulary for this feature.
outputs[key] = transform.string_to_int(
inputs[key], top_k=taxi.VOCAB_SIZE,
num_oov_buckets=taxi.OOV_SIZE)
for key in taxi.BUCKET_FEATURE_KEYS:
outputs[key] = transform.bucketize(inputs[key],

Beam Demo Time!!!!!
● TFT!!!!!! So amazing!!!!!!!!!
○ Want to move to SF and raise a series A? Pay attention :p
● Based on my testing on Saturday there is a 1 in 3 chance this will hard lock
my computer
○ But that’s OK, we can reboot and take questions while I bounce :)
● That’s your friendly reminder not to run any of this in production (yet)
● The demo code is forked from axelmagn/model-analysis which is forked from
tensorflow/model-analysis and runs on master of Apache Beam w/Flink
○ The example does something… possible questionable with the evaluation dataset currently,
but #todofixmelater

Ok now what?
● Integrate this into your model serving pipeline of choice
○ Don’t have one or open to change? Checkout TFMA which can directly serve it
● There’s a guide (it doesn’t show Flink because not released yet) but steps are
similar
○ But you’re not using this in production today anyways?
○ Right?
Nick Perla

(Optional) Second Beam Demo Time!!!!!
● Word count!!!!!! So amazing!!!!!!!!!
● Based on my testing on Saturday there is a 2 in 3 chance this will hard lock
my computer
● That’s your friendly reminder not to run any of this in production
● Demo shell script of fun (go only) & python + go

What do the rest of the systems do?
● Spoiler: mostly it’s not better
○ Although it tends to be more finished
○ Sometimes it's different
● Different tradeoffs, maybe better for your use case but all tradeoffs
Kate Neilan

A quick detour into PySpark’s internals
+ + JSON
TimOve

PySpark
● The Python interface to Spark
● Same general technique used as the bases for the C#, R, Julia, etc.
interfaces to Spark
● Fairly mature, integrates well-ish into the ecosystem, less a Pythonrific API
● Has some serious performance hurdles from the design

So what does that look like?
Driver
py4j
Worker 1
Worker K
pipe
pipe

And in flink….
Driver
custom
Worker 1
Worker K
mmap
mmap

So how does that impact Py[X]
forall X in {Big Data}-{Native Python Big Data}
● Double serialization cost makes everything more
expensive
● Python worker startup takes a bit of extra time
● Python memory isn’t controlled by the JVM - easy to go
over container limits if deploying on YARN or similar
● Error messages make ~0 sense
● Dependency management makes limited sense
● features aren’t automatically exposed, but exposing
them is normally simple

TensorFlowOnSpark, everyone loves mnist!
cluster = TFCluster.run(sc, mnist_dist_dataset.map_fun, args,
args.cluster_size, num_ps, args.tensorboard,
TFCluster.InputMode.SPARK)
if args.mode == "train":
cluster.train(dataRDD, args.epochs)
Lida

The “future”*: faster interchange
● By future I mean availability today but running it in production is “adventurous”
● Unifying our cross-language experience
○ And not just “normal” languages, CUDA counts yo
Tambako The Jaguar

Andrew Skudder
*Arrow: Spark 2.3 and beyond & GPUs & R & Python & ….
* *

What does the future look like?*
*Source: https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html.
*Vendor
benchmark.
Trust but verify.

Arrow (a poorly drawn big data view)
Logos trademarks of their respective projects
Juha Kettunen
*ish

Rewriting your code because why not
spark.catalog.registerFunction(
"add", lambda x, y: x + y, IntegerType())
=>
add = pandas_udf(lambda x, y: x + y, IntegerType())
Jennifer C.

And we can do this in TFOnSpark*:
unionRDD.foreachPartition(TFSparkNode.train(self.cluster_info,
self.cluster_meta, qname))
Will Transform Into something magical (aka fast but unreliable)
on the next slide!
Delaina Haslam

Which becomes
train_func = TFSparkNode.train(self.cluster_info,
self.cluster_meta, qname)
@pandas_udf("int")
def do_train(inputSeries1, inputSeries2):
# Sad hack for now
modified_series = map(lambda x: (x[0], x[1]),
zip(inputSeries1, inputSeries2))
train_func(modified_series)
return pandas.Series([0] * len(inputSeries1))
ljmacphee

And this now looks like:
Logos trademarks of their respective projects
Juha Kettunen
*ish

TFOnSpark Possible vNext+1?
● Avoid funneling the data through Python native types
○ For now the Spark Arrow UDFS aren’t perfect for this
○ But we can (and are) improving them
● mmapped Arrow?
● Skip Python on the workers handling data entirely (idk I’m lazy so probably
not)
Renars

References
● TFMA + TFT example guide -
https://www.tensorflow.org/tfx/model_analysis/examples/chicago_taxi
● Apache Beam github repo (w/early alpha portable Flink support)-
https://beam.apache.org/
● TFMA Example fork for use w/Beam on Flink -
● TensorFlowOnSpark -https://github.com/yahoo/TensorFlowOnSpark
● Spark Deep Learning Pipelines - https://github.com/databricks/spark-deep-
learning
● flink-tensorflow - https://github.com/FlinkML/flink-tensorflow
● TF.Transform - https://github.com/tensorflow/transform
● Beam portability design: https://beam.apache.org/contribute/portability/
● Beam on Flink + portability https://issues.apache.org/jira/browse/BEAM-2889
PROR. Crap Mariner

k thnx bye :)
If you care about Spark testing and
don’t hate surveys:
http://bit.ly/holdenTestingSpark
I need to give a testing talk next
month, help a “friend” out.
Will tweet results
“eventually” @holdenkarau
Do you want more realistic
benchmarks? Share your UDFs!
http://bit.ly/pySparkUDF
Pssst: Have feedback on the presentation? Give me a
shout (holden@pigscanfly.ca) if you feel comfortable doing
so :)
Give feedback on this presentation
http://bit.ly/holdenTalkFeedback

What’s the rest of big data outside the JVM
look like?
Most of the tools are built in the JVM, so how do we play together?
● Pickling, Strings, JSON, XML, oh my!
● Unix pipes
● Sockets
What about if we don’t want to copy the data all the time?
● Or standalone “pure”* re-implementations of everything
○ Reasonable option for things like Kafka where you would have the I/O regardless.
○ Also cool projects like dask (pure python) -- but hard to talk to existing ecosystem
David Brown

Hadoop “streaming” (Python/R)
● Unix pipes!
● Involves a data copy, formats get sad
● But the overhead of a Map/Reduce task is pretty high anyways...
Lisa Larsson

Kafka: re-implement all the things
● Multiple options for connecting to Kafka from outside of the JVM (yay!)
● They implement the protocol to talk to Kafka (yay!)
● This involves duplicated client work, and sometimes the clients can be slow
(solution, FFI bindings to C instead of Java)
● Buuuut -- we can’t access all of the cool Kafka business (like Kafka Streams)
and features depend on client libraries implementing them (easy to slip below
parity)
Smokey Combs

Dask: a new beginning?
● Pure* python implementation
● Provides real enough DataFrame interface for distributed data
● Also your standard-ish distributed collections
● Multiple backends
● Primary challenge: interacting with the rest of the big data ecosystem
○ Arrow & friends might make this better with time too, buuut….
● See https://dask.pydata.org/en/latest/ &
http://dask.pydata.org/en/latest/spark.html
Lisa Zins

Simplifying training deep and serving learning models with big data in python using tensorflow - PyData Berlin

Recommended

More Related Content

What's hot (20)

Similar to Simplifying training deep and serving learning models with big data in python using tensorflow - PyData Berlin (20)

Recently uploaded (20)

Simplifying training deep and serving learning models with big data in python using tensorflow - PyData Berlin

Editor's Notes