Embracing a Taxonomy of Types to Simplify Machine Learning with Leah McGuire

Leah McGuire
Principal Member of Technical Staﬀ, Salesforce Einstein
lmcguire@salesforce.com
@leahmcguire
Types, Types, Types!
Embracing a hierarchy of types to simplify machine
learning

Let’s make sure you are in the right talk
 What I am going to talk about:
• What does machine learning mean at Salesforce
• Problems in machine learning for business to business (B2B)
companies
• Automating machine learning and how our AutoML library (Optimus
Prime) works
• The utility of having strongly typed features in AutoML
• What we have learned and what we are planning

Salesforce and Machine
Learning

The Problem
 For the majority of businesses, data science is out of reach

Sales Cloud Einstein
Predictive Lead Scoring
Opportunity Insights
Automated Activity Capture
Building World’s Smartest CRM
Commerce Cloud
Einstein
Product Recommendations
Predictive Sort
Commerce Insights
App Cloud Einstein
Heroku + PredictionIO
Predictive Vision Services
Predictive Sentiment Services
Predictive Modeling Services
Service Cloud Einstein
Recommended Case Classification
Recommended Responses
Predictive Close Time
Marketing Cloud Einstein
Predictive Scoring
Predictive Audiences
Automated Send-time Optimization
Community Cloud Einstein
Recommended Experts, Articles & Topics
Automated Service Escalation
Newsfeed Insights
Analytics Cloud Einstein
Predictive Wave Apps
Smart Data Discovery
Automated Analytics & Storytelling
IoT Cloud Einstein
Predictive Device Scoring
Recommend Best Next Action
Automated IoT Rules Optimization

Machine learning workﬂows
And how much more complicated they get for B2B

Feature
Engineering
Model
Training
Model A
Model B
Model C
Model
Evaluation
 What Kaggle would lead us to believe
Building a machine learning model

Real-life ML
 Building a ML model workﬂow
ETL
Model
Evaluation
Feature
Engineering
Scoring
Model
Training
Model A
Model B
Model C
Deployment

 Over and over again
ETL
Model
Evaluation
Feature
Engineering
Scoring
Model
Trainin
g
Model
A
Model
B
Model
C
Deployment
ETL
Model
Evaluation
Feature
Engineering
Scoring
Model
Trainin
g
Model
A
Model
B
Model
C
Deployment
ETL
Model
Evaluation
Feature
Engineering
Scoring
Model
Trainin
g
Model
A
Model
B
Model
C
Deployment

We can’t build one global model
• Privacy concerns
•  Customers don’t want data cross-
pollinated
• Business Use Cases
•  Industries are very different
•  Processes are different
• Platform customization
•  Ability to create custom fields and
objects
• Scale, Automation,
•  Ability to create

 Over and over again
M
o
d
e
l
E
v
a
l
u
a
t
i
o
n
F
e
a
t
u
r
e
E
n
g
i
n
e
e
r
i
n
g
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
M
o
d
e
l
E
v
a
l
u
a
t
i
o
n
F
e
a
t
u
r
e
E
n
g
i
n
e
e
r
i
n
g
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
M
o
d
e
l
E
v
a
l
u
a
t
i
o
n
F
e
a
t
u
r
e
E
n
g
i
n
e
e
r
i
n
g
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
M
o
d
e
l
E
v
a
l
u
a
t
i
o
n
F
e
a
t
u
r
e
E
n
g
i
n
e
e
r
i
n
g
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
ET
L
Model
Evaluati
on
Feature
Enginee
ring
Scoring
Mod
el
Trai
ning
Mod
el A
Mod
el B
Mod
el C
Deploy
ment
M
o
d
el
E
v
al
u
at
io
n
F
e
at
ur
e
E
n
gi
n
e
er
in
g
S
c
o
ri
n
g
M
o
d
e
l
T
r
a
i
n
i
n
g
M
o
d
e
l
A
M
o
d
e
l
B
M
o
d
e
l
C
D
e
pl
o
y
m
e
nt
M
o
d
e
l
E
v
a
l
u
a
t
i
o
n
F
e
a
t
u
r
e
E
n
g
i
n
e
e
r
i
n
g
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
M
o
d
e
l
E
v
a
l
F
e
a
t
u
r
e
E
n
g
i
n
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
M
o
d
e
l
E
v
a
l
u
a
t
i
o
n
F
e
a
t
u
r
e
E
n
g
i
n
e
e
r
i
n
g
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
M
o
d
e
l
E
v
F
e
a
t
u
r
e
E
n
S
c
o
r
i
n
g
D
e
p
l
o
y
m
M
o
d
e
l
E
v
a
l
u
F
e
a
t
u
r
e
E
n
g
i
n
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
M
o
a
F
e
a
t
u
r
S
c
o
D
e
p
Mo
del
Ev
alu
ati
on
Fe
atu
re
En
gin
eer
ing
Sc
ori
ng
M
o
d
e
l
T
r
a
i
n
i
n
g
M
o
d
e
l
A
M
o
d
e
l
B
M
o
d
e
l
C
De
plo
ym
ent
M
o
d
e
l
E
v
a
l
u
a
t
i
o
F
e
a
t
u
r
e
E
n
g
i
n
e
e
r
i
n
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
M
o
d
e
l
E
v
a
l
u
a
t
i
o
n
F
e
a
t
u
r
e
E
n
g
i
n
e
e
r
i
n
g
S
c
o
r
i
n
g
D
e
p
l
o
y
m
e
n
t
M
o
d
e
l
E
v
F
e
a
t
u
r
e
E
n
g
S
c
o
r
i
n
g
D
e
p
l
o
y
m

Automating machine learning
Enter Einstein (and Optimus Prime)

•  ML is not magic, just statistics –
generalizing examples
•  But there is a ‘black art’ to producing
good models
•  Input data needs to be combined, ﬁltered,
cleaned etc.
•  Producing the best features for your model
takes time
•  You can’t just throw a ml algorithm at your raw
data and expect good results
Turning a black art into a paint by number kit.

Keep it DRY (don’t repeat yourself) and DRO (don’t repeat
others)
• The Spark ML pipeline (estimator, transformer) model is nice
• The lack of types in Spark is not
• Want to use more than Spark ML
• Declarative and intuitive syntax – for both
workﬂow generation and developers
• Typed reusable operations
• Multitenant application support
• All built in scala
Optimus Prime - A library to develop reusable, modular and typed ML workﬂows

Simple interchangeable parts
 val featureVector = Seq(pClass, name, gender, age, sibSp, parch, ticket, cabin, embarked).vectorize()
 val (pred, raw, prob) = featureVector.check(survived).classify(survived)
 val workflow = new OpWorkflow().setResultFeatures(pred).setDataReader(titanicReader)
 In a declarative type safe syntax
Stages
Features
transformed with
produce
Transformers Estimators
fitted into
ReadersWorkflows
input data
materialized by

Automating typed feature
engineering and modeling
(with Optimus Prime)

Features are given a type on creation
 val gender = FeatureBuilder.Categorical[Titanic]
.extract(d => Option(d.getGender).toSet[String]).asPredictor
• Features are strongly typed
• Each stage takes speciﬁc input type(s) and returns a speciﬁc output type(s)
 Death to runtime errors!
StagesFeatures
produce

Creating a workﬂow DAG with features
• Features point to a
column of data
• The type of the feature
determines which stages
can act on it
pred
prob
rawPred
survived
featureVector
gender age name
genderPivot title

Creating a workﬂow DAG with features
• When a stage acts on a
feature it produces a new
feature (or features)
• Keep on manipulating
features until you get
your goal
pred
prob
rawPred
survived
featureVector
gender age name
genderPivot title
Pivot Title Regex
Combine
Model

Done manipulating your features? Make them.
Stages
Features
transformed with
produce
Transformers Estimators
fitted
into
ReadersWorkflows
input data
materialized by
• Once you make your final feature you have the full DAG
• Features are materialized by the workflow
• Initial data into the workflow provided by the reader

Using types to automate feature engineering
 val featureVector = Seq(pClass, name, gender, age, sibSp, parch, ticket, cabin, embarked).vectorize()
• Each feature is mapped to an appropriate .vectorize() stage based on its type
•  gender (a Categorical) and age (a Real) are automatically assigned to diﬀerent stages
• You also have an option to do the exact type safe manipulations you want
•  age can undergo special transformations if desired
•  val ageBuckets =age.bucketize(buckets(0, 10, 20, 40, 100))
•  val featureVector = Seq(pClass, name, gender, ageBuckets, sibSp, parch, ticket, cabin,
embarked).vectorize()

Show me the types!
FeatureType
OPNumeric OPCollection
OPSet
OPSortedSet
OPList
NonNullable
Text
Email
Base64
Phone
ID
URL
ComboBox
PickList
TextArea
OPVector OPMap
BinaryMap
IntegralMap
RealMap
CategoricalMap
OrdinalMap
DateList
DateTimeList
Integral
Real
Binary
Percent
Currency
Date
DateTime Categorical
MultiPickList
Note: all the types are assumed to be nullable, unless NonNullable trait is mixed - https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/field_types.htm
Ordinal
TextMap
Legend: - inheritance, bold - abstract class, italic - trait, normal - concrete class
...
Optimus Prime Type Hierarchy
TextList
City
Street
Country
Postal Code
Location
State
Geolocation
StateMap

Take the types away!!
• Sometimes a type is all you have
• Hierarchy allows both very speciﬁc
and very general stages
• Type safety for production saves a
lot of headaches
 Why would we make this monstrosity??
FeatureType
OPNumeric OPCollection
OPSet
OPSortedSet
OPList
NonNullable
Text
Email
Base64
Phone
ID
URL
ComboBox
PickList
TextArea
OPVector OPMap
BinaryMap
IntegralMap
RealMap
CategoricalMap
OrdinalMap
DateList
DateTimeList
Integral
Real
Binary
Percent
Currency
Date
DateTime Categorical
MultiPickList
Note: all the types are assumed to be nullable, unless NonNullable trait is mixed - https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/field_types.htm
Ordinal
TextMap
Legend: - inheritance, bold - abstract class, italic - trait, normal - concrete class
...
Optimus Prime Type Hierarchy
TextList
City
Street
Country
Postal Code
Location
State
Geolocation
StateMap

Sanity Checking – the stage that checks your features
• Check data quality before modeling
• Label leakage
• Features have acceptable ranges
• The feature types allow much better checks
val checkedVector = featureVector.check(survived)

Model Selection Stage - Resampling, Hyper-parameter
Tuning, Comparing Models
• Many possible models for each class of
problem
• Many hyper parameters for each type of
model
• Finding the right model for THIS dataset
makes a huge diﬀerence
val (pred, raw, prob) = checkedFeatureVector.classify(survived)

Types can save us
 And if you don’t believe me take a look at the code
val featureVector = Seq(pClass, name, gender, age, sibSp, parch, ticket, cabin, embarked).vectorize()
val (pred, raw, prob) = featureVector.check(survived).classify(survived)
val workﬂow = new OpWorkﬂow().setResultFeatures(pred).setDataReader(titanicReader)

Types can save us
val featureVector = Seq(pClass, name, gender, age, sibSp, parch, ticket, cabin, embarked).vectorize()
val (pred, raw, prob) = featureVector.check(survived).classify(survived)
val workﬂow = new OpWorkﬂow().setResultFeatures(pred).setDataReader(titanicReader)
def addFeatures(df: DataFrame): DataFrame = {!
// Create a new family size field := siblings + spouses + parents + children + self!
val familySizeUDF = udf { (sibsp: Double, parch: Double) => sibsp + parch + 1 }!
!
df.withColumn("fsize", familySizeUDF(col("sibsp"), col("parch"))) // <-- full freedom to overwrite !
}!
!
def fillMissing(df: DataFrame): DataFrame = {!
// Fill missing age values with average age!
val avgAge = df.select("age").agg(avg("age")).collect.first()!
!
// Fill missing embarked values with default "S" (i.e Southampton)!
val embarkedUDF = udf{(e: String)=> e match { case x if x == null || x.isEmpty => "S"; case x => x}}!
!
df.na.fill(Map("age" -> avgAge)).withColumn("embarked", embarkedUDF(col("embarked")))!
}!

Types can save us
// Modify the dataframe!
val allData = fillMissing(addFeatures(rawData)).cache() // <-- need to remember about caching!
// Split the data and cache it!
val Array(trainSet, testSet) = allData.randomSplit(Array(0.75, 0.25)).map(_.cache())!
!
// Prepare categorical columns!
val categoricalFeatures = Array("pclass", "sex", "embarked")!
val stringIndexers = categoricalFeatures.map(colName =>!
new StringIndexer().setInputCol(colName).setOutputCol(colName + "_index").fit(allData)!
)!
!
// Concat all the feature into a numeric feature vector!
val allFeatures = Array("age", "sibsp", "parch", "fsize") ++ stringIndexers.map(_.getOutputCol)!
!
val vectorAssembler = new VectorAssembler().setInputCols(allFeatures).setOutputCol("feature_vector”)!
!
// Prepare Logistic Regression estimator!
val logReg = new LogisticRegression().setFeaturesCol("feature_vector").setLabelCol("survived”)!
!
// Finally build the pipeline with the stages above!
val pipeline = new Pipeline().setStages(stringIndexers ++ Array(vectorAssembler, logReg))!

Types can save us
// Cross validate our pipeline with various parameters!
val paramGrid =!
new ParamGridBuilder()!
.addGrid(logReg.regParam, Array(1, 0.1, 0.01))!
.addGrid(logReg.maxIter, Array(10, 50, 100))!
.build()!
!
val crossValidator =!
new CrossValidator()!
.setEstimator(pipeline) // <-- set our pipeline here!
.setEstimatorParamMaps(paramGrid)!
.setEvaluator(new BinaryClassificationEvaluator().setLabelCol("survived"))!
.setNumFolds(3)!
!
// Train the model & compute scores !
val model: CrossValidationModel = crossValidator.fit(trainSet)!
val scores: DataFrame = model.transform(testSet)!
!
// Save the model for later use!
model.save("/models/titanic-model.ml")!
!

Where are we going and what
have we learned

Key takeaways
• ML for B2B is a whole other beast
• Spark ML is great, but it needs type safety
• Simple and intuitive syntax saves you trouble down the road
• Types in ML are incredibly useful
• Scala has all the relevant facilities to provide the above
• Modularity and reusability is the key

Going forward with Optimus Prime
• Going beyond Spark ML for algorithms and small scale
• Making everything smarter (feature eng, sanity checking, model selection)
• Template generation
• Improvements to developer interface

If You’re Curious …
einstein-recruiting@salesforce.com

Embracing a Taxonomy of Types to Simplify Machine Learning with Leah McGuire

Recommended

More Related Content

What's hot (20)

Similar to Embracing a Taxonomy of Types to Simplify Machine Learning with Leah McGuire (20)

More from Databricks (20)

Recently uploaded (20)

Embracing a Taxonomy of Types to Simplify Machine Learning with Leah McGuire