Kerberizing spark. Spark Summit east

Jorge López-Malla Matute
INDEX
jlopezm@stratio.com
Abel Rincón Matarranz
arincon@stratio.com
Kerberos
● Introduction
● Key concepts
● Workflow
● Impersonation
1
3
Use Case
● Definition
● Workflow
● Crossdata in production
2
4Stratio Solution
● Prerequirement
● Driver side
● Executor Side
● Final result
Demo time
● Demo
● Q&A

Presentation
Presentation
JORGE LÓPEZ-MALLA
After working with traditional
processing methods, I started to
do some R&S Big Data projects
and I fell in love with the Big Data
world. Currently i’m doing some
awesome Big Data projects
and tools at Stratio.
SKILLS

Presentation
Presentation
ABEL RINCÓN MATARRANZ
SKILLS

Kerberos
Kerberos
• What is Kerberos
○ Authentication protocol / standard / service
■ Safe
■ Single-sign-on
■ Trust based
■ Mutual authentication

Kerberos key concepts
Kerberos
• Client/Server → Do you need an explanation???
• Principal → Identify a unique client or service
• Realm → Identify a environment, company, domain …
○ DEMO.EAST.SUMMIT.SPARK.ORG
• KDC → Actor who manages the tickets
• TGT → Ticket which has the client session
• TGS → Ticket which has the client-service session

Kerberos Workflow
Kerberos
1. Client retrieve principal and secret
2. Client performs a TGT request
3. KDC returns TGT
4. Client request a TGS with the TGT
5. KDC returns the TGS
6. El cliente request a service session using
the TGS
7. Service establish a secure connection
directly with the client

Kerberos workflow 2
Kerberos
Client
Service
Backend
AS / KDC
Principal → User1
TGT (user1)
TGS → user1-service1
(tgsUS)
Principal → Service1
TGT (Service1)
TGS → service1-backend1 (tgsSB)
Principal → backend1
TGT (backend1)
tgsUS
tgsSB
user1
service1

Kerberos workflow - Impersonation
Kerberos
Client
Service
Backend
AS / KDC
Principal → User1
TGT (user1)
TGS → user1-service1
(tgsUS)
Principal → Service1
TGT (Service1)
TGS → service1-backend1 (tgsSB)
Principal → backend1
TGT (backend1)
tgsUS
tgsSB
user1
service1user1

Use Case
• Stratio Crossdata is a distributed framework and a fast and general-purpose
computing system powered by Apache Spark
• Can be used both as library and as a server.
• Crossdata Server: Provides a multi-user environment to SparkSQL, giving a
reliable architecture with high-availability and scalability out of the box
• To do so it use both native queries and Spark
• Crossdata Server had a unique long time SparkContext to execute all its Sparks
queries
• Crossdata can use YARN, Mesos and Standalone as a resource manager
Use Case

Crossdata as Server
sql> select * from table1
Crossdata shell
Master
Worker-1 Worker-2
Executor-0 Executor-1
Task-0 Task-1
HDFS
Crossdata server
(Spark Driver)
Crossdata server
sql> select * from table1
-------------------------
|id | name |
-------------------------
|1 | John Doe |
Use Case
Kerbe
ros

• Projects in production needs runtime impersonation to be compliance the
AAA(Authorization, Authentication and Audit) at the storage.
• Crossdata allows several users per execution
• Neither of the Sparks resource managers allows us to impersonate in runtime.
• Evenmore Standalone as resource manager does not provide any Kerberos
feature.
Crossdata in production
Use Case

Prerequirement
Stratio solution
• Keytab have to be accessible in all the cluster machines
• Keytab must provide proxy grants
• Hadoop client configuration located in the cluster
• Each user, both proxy and real, must have a home in HDFS

Introduction
Stratio solution
• Spark access to the storage system both in the Driver and the Executors.
• In the Driver side both Spark Core and SparkSQL will access to the storage
system.
• Executors will always access via Task.
• As Streaming use the same classes than SparkCore or SparkSQL the same
solution will be usable by Streaming jobs

KerberosUser (Utils)
object KerberosUser extends Logging with UserCache {
def setProxyUser(user: String): Unit = proxyUser = Option(user)
def getUserByName(name: Option[String]): Option[UserGroupInformation] = {
if (getConfiguration.isDefined) {
userFromKeyTab(name)
}
else None
}
private def userFromKeyTab(proxyUser: Option[String]): Option[UserGroupInformation] = {
if (realUser.isDefined) realUser.get.checkTGTAndReloginFromKeytab()
(realUser, proxyUser) match {
case (Some(_), Some(proxy)) => users.get(proxy).orElse(loginProxyUser(proxy))
case (Some(_), None) => realUser
case (None, None) => None
}
}
private lazy val getConfiguration: Option[(String, String)] = {
val principal = env.conf.getOption("spark.executor.kerberos.principal")
val keytab = env.conf.getOption("spark.executor.kerberos.keytab")
(principal, keytab) match {
case (Some(p), Some(k)) => Option(p, k)
case _ => None
}
}
Configuration
setting proxy user
(Global)
Choose between real or
proxy user
Stratio solution
Public Method retrieve
user

Wrappers (Utils)
def executeSecure[U, T](proxyUser: Option[String],
funct: (U => T),
inputParameters: U): T = {
KerberosUser.getUserByName(proxyUser) match {
case Some(user) => {
user.doAs(new PrivilegedExceptionAction[T]() {
@throws(classOf[Exception])
def run: T = {
funct(inputParameters)
}
})
}
case None => {
funct(inputParameters)
}
}
}
def executeSecure[T](exe: ExecutionWrp[T]): T = {
KerberosUser.getUser match {
case Some(user) => {
user.doAs(new PrivilegedExceptionAction[T]() {
@throws(classOf[Exception])
def run: T = {
exe.value
}
})
}
case None => exe.value
}
}
class ExecutionWrp[T](wrp: => T) {
lazy val value: T = wrp
}
Stratio solution

Driver Side
Stratio Solution
abstract class RDD[T: ClassTag](
@transient private var _sc: SparkContext,
@transient private var deps: Seq[Dependency[_]]
) extends Serializable with Logging {
...
/**
* Get the array of partitions of this RDD, taking into account whether the
* RDD is checkpointed or not.
*/
final def partitions: Array[Partition] = {
checkpointRDD.map(_.partitions).getOrElse {
if (partitions_ == null) {
partitions_ = KerberosFunction.executeSecure(new ExecutionWrp(getPartitions))
partitions_.zipWithIndex.foreach { case (partition, index) =>
require(partition.index == index,
s"partitions($index).partition == ${partition.index}, but it should equal $index")
}
}
partitions_
}
}
Wrapping parameterless
method

class PairRDDFunctions[K, V](self: RDD[(K, V)])
(implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null)
...
def saveAsHadoopDataset(conf: JobConf): Unit = self.withScope {
// Rename this as hadoopConf internally to avoid shadowing (see SPARK-2038).
val internalSave: (JobConf => Unit) = (conf: JobConf) => {
val hadoopConf = conf
val outputFormatInstance = hadoopConf.getOutputFormat
val keyClass = hadoopConf.getOutputKeyClass
val valueClass = hadoopConf.getOutputValueClass
...
val writeToFile = (context: TaskContext, iter: Iterator[(K, V)]) => {
….
}
self.context.runJob(self, writeToFile)
writer.commitJob()
}
KerberosFunction.executeSecure(internalSave, conf)
}
Driver Side
Stratio Solution
Hadoop Datastore
RDD save function
Inside wrapper function
that will run in the cluster
Kerberos authentified
save function

Driver Side
class InMemoryCatalog(
conf: SparkConf = new SparkConf,
hadoopConfig: Configuration = new Configuration)
override def createDatabase(
dbDefinition: CatalogDatabase,
ignoreIfExists: Boolean): Unit = synchronized {
def inner: Unit = {
...
val location = new Path(dbDefinition.locationUri)
val fs = location.getFileSystem(hadoopConfig)
fs.mkdirs(location)
} catch {
case e: IOException =>
throw new SparkException(s"Unable to create database ${dbDefinition.name} as failed " +
s"to create its directory ${dbDefinition.locationUri}", e)
}
catalog.put(dbDefinition.name, new DatabaseDesc(dbDefinition))
}
}
KerberosFunction.executeSecure(KerberosUser.principal, new ExecutionWrp(inner))
}
Stratio Solution
Spark create a
directory in HDFS

* Interface used to load a [[Dataset]] from external storage systems (e.g. file systems,
...
class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging {
...
def load(): DataFrame = {
load(Seq.empty: _*) // force invocation of `load(...varargs...)`
}
...
def load(paths: String*): DataFrame = {
val proxyuser = extraOptions.get("user")
if (proxyuser.isDefined) KerberosUser.setProxyUser(proxyuser.get)
val dataSource = KerberosFunction.executeSecure(proxyuser,
DataSource.apply,
sparkSession,
source,
paths, userSpecifiedSchema, Seq.empty, None, extraOptions.toMap)
val baseRelation = KerberosFunction.executeSecure(proxyuser, dataSource.resolveRelation, false)
KerberosFunction.executeSecure(proxyuser, sparkSession.baseRelationToDataFrame, baseRelation)
}
Driver Side
Stratio Solution
get user from dataset
options
Method for load data from
sources without path
obtaining baseRelation
from datasource

* Interface used to write a [[Dataset]] from external storage systems (e.g. file systems,
...
class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
...
/**
* Saves the content of the [[DataFrame]] as the specified table.
...
def save(): Unit = {
assertNotBucketed("save")
...
val maybeUser = extraOptions.get("user")
def innerWrite(modeData: (SaveMode, DataFrame)): Unit = {
val (mode, data) = modeData
dataSource.write(mode, data)
}
if (maybeUser.isDefined) KerberosUser.setProxyUser(maybeUser.get)
KerberosFunction.executeSecure(maybeUser, innerWrite, (mode, df))
}
Driver Side
Stratio Solution
get user from dataset
options
Method for save data in
external sources
Wrapping save
execution

class DAGScheduler(...){
…
…
KerberosUser.getMaybeUser match {
case Some(user) => properties.setProperty("user", user)
case _ =>
}
...
val tasks: Seq[Task[_]] = try {
stage match {
case stage: ShuffleMapStage =>
partitionsToCompute.map { id =>
...
new ShuffleMapTask(stage.id, stage.latestInfo.attemptId,
taskBinary, part, locs, stage.latestInfo.taskMetrics, properties)
...
new ResultTask(stage.id, stage.latestInfo.attemptId,
taskBinary, part, locs, id, properties, stage.latestInfo.taskMetrics)
}
...
Driver Side
Stratio Solution

private[spark] abstract class Task[T](
val stageId: Int,
val stageAttemptId: Int,
val partitionId: Int,
// The default value is only used in tests.
val metrics: TaskMetrics = TaskMetrics.registered,
@transient var localProperties: Properties = new Properties) extends Serializable {
...
final def run(
...
try {
val proxyUser =
Option(Executor.taskDeserializationProps.get().getProperty("user"))
KerberosFunction.executeSecure(proxyUser, runTask, context)
} catch {
…
def runTask(context: TaskContext): T
Executor Side
Stratio Solution
properties load in
Driver side
get proxy user and
wrapped execution
method implemented by
Task subclasses

• Merge this code in Apache Spark (SPARK-16788)
• Pluggable authorization
• Pluggable secret management (why always use Hadoop delegation tokens?)
• Distributed cache.
• ...
Next Steps
Next Steps

Kerberizing spark. Spark Summit east

Recommended

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Kerberizing spark. Spark Summit east (20)

More from Jorge Lopez-Malla (8)

Recently uploaded (20)

Kerberizing spark. Spark Summit east