Apache Spark and Scala

Module 4: Functional Programming in Scala

© 2015 BlueCamphor Technologies (P) Ltd.

 Course Topics

Module 1 Module 2 Module 3 Module 4

Getting Started / Scala – Essentials and Introducing Traits and Functional Programming
Introduction to Scala Deep Dive OOPS in Scala in Scala

Module 5 Module 6 Module 7 Module 8

Spark and Big Data Advanced Spark Understanding RDDs Shark, SparkSQL and
Concepts Project Discussion

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 2

 Session Objectives
In this session, you will learn about

ᗍ Traits as Mixins
ᗍ Functional Programming
ᗍ Paradigms of Functional Programming
ᗍ Higher Order Functions
ᗍ Currying
ᗍ Closures
ᗍ Anonymous Blocks,
ᗍ Implicit Function Parameters
ᗍ Call by Name
ᗍ Call by Value

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 3

 Why No Multiple Inheritance?

ᗍ Multiple inheritance works when parent classes have nothing in common

ᗍ Scala, like Java doesn’t allow a class to inherit from multiple classes
ᗍ Problem arises if they have common functionality or fields

Example:

Class Person {
def id: String ...
.....
}

Class Employee{
def id: String ...
...
}

//Assume we allow multiple inheritance like:

class Analyst extends Employee, Person{

...
}

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 4

 Why No Multiple Inheritance? (Cont’d)

ᗍ Now, Sample class has two id methods, which shall be used?

ᗍ The same is true for the fields
ᗍ This problem is famously called as Diamond Inheritance problem, wherein, the child class would be unable to
distinguish between the common members of superclasses
ᗍ In Java, this problem is solved by the concept of interfaces, where the interfaces have only the abstract methods
ᗍ Scala has the concept of Traits instead of interfaces
ᗍ Scala traits, unlike Java interfaces could have the abstract and concrete methods

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 5

 Traits as Interfaces
ᗍ Similar to interfaces in Java, traits are used to define object types by specifying the signature of the supported
methods
ᗍ The syntax is the same syntax we use when defining a class. The only difference is using the 'trait' keyword instead
of 'class'

Example:

ᗍ Unlike Java, Scala allows traits to be partially implemented; i.e. it is possible to define default implementations for
some methods
ᗍ Methods need not be declared as abstract
ᗍ An unimplemented method is automatically assumed as abstract method

-------> Abstract method

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 6

 Traits as Interfaces (Cont’d)

ᗍ Once a trait is defined, it can be mixed in to a class using either the keyword extends or the keyword with
ᗍ We can use methods inherited from a trait just as any method inherited from a super class
ᗍ Once a trait is defined we get a new type, similarly to defining a new class

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 7

 Traits with Concrete Implementation
Trait methods can be concrete also

The concrete trait method provides log method with its implementation

Below is the example of its usage:

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 8

 Traits with Concrete Implementation (Cont’d)

ᗍ The subclass picks up the concrete implementation from the trait

ᗍ Thus the ConcreteTrait functionality is mixed in with Ranker class
ᗍ But whenever the trait implementation changes, all the mixed in classes needs to be re-compiled

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 9

 Objects with Traits

Traits could be added with individual objects while constructing them

Example:
We’ll use Logged trait of standard Scala library:

What should be the expected behaviour?

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 10

 Objects with Traits (Cont’d)
Nothing gets logged! As the extended trait didn’t have any implementation

Now, let’s extend the trait:

Now we can add this trait while constructing a new object:

ᗍ Now when Top method is executed, the else part of the Logger1 is invoked and displays the message
ᗍ For the class-private field, private getter and setter are generated
ᗍ Thus we get the flexibility of attaching use case specific loggers!

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 11

 Layered Traits

ᗍ Multiple traits can be added to a class or object in Scala

ᗍ In such case, the traits are invoked always starting from the last
ᗍ Extending multiple traits could be useful for the cases when a value needs to be transformed in stages
Example:

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 12

 Layered Traits
Now, assume that we want to truncate long message like:

ᗍ Note that each log method calls a super.log

ᗍ In traits, the keyword trait doesn’t have the same meaning as it does with classes
ᗍ Here super.log calls the next trait in the trait hierarchy
ᗍ Hierarchy depends upon the order in which traits are added
ᗍ Traits are processed starting with last one

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 13

 Traits Construction Order
Traits are constructed in following order:

ᗍ The superclass constructor is called first

ᗍ Trait constructors are executed after the superclass constructor but before the class constructor
ᗍ Traits are constructed from left to right
ᗍ Within each trait, the parent gets constructed first
ᗍ If multiple traits share the same parent and if the parent has already been constructed, it is not re-constructed
ᗍ After all the traits are constructed, the subclass is constructed

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 14

 Check your Understanding – 1

Traits construction order is decided by their they inheritance

a. True
b. False

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 15

 Check your Understanding – Solution

Traits construction order is decided by their they inheritance

a. True
b. False

False

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 16

 Functional Programming

ᗍ Apart from being a pure object oriented language, Scala is also a functional programming language
ᗍ Functional programming is driven by mainly two ideas:
• First main idea is that functions are first class values. They are treated just like any other type, say String,
Integer etc. So functions can be used as arguments, could be defined in other functions
• The second main idea of functional programming is that the operations of a program should map input
values to output values rather than change data in place. This results in the immutable data structures

Scala supports both, immutable and mutable data structures

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 17

 Higher Order Functions

ᗍ Functional languages treat functions as first-class values

ᗍ This means that, like any other value, a function can be passed as a parameter and returned as a result
ᗍ This provides a flexible way to compose programs
ᗍ Functions that take other functions as parameters or that return functions as results are called higher order
functions

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 18

 Examples

Take the sum of the integers between a and b

Take the sum of the squares of all the integers between a and b:

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 19

 Examples (Cont’d)
Take the sum of the factorials of all the integers between a and b:

These are special cases of below for different values of f

b
Σf(n)
n=a
Can we factor out the common pattern?

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 20

 Summing with Higher Order Functions
Let’s define:

We can then write:

Where

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 21

 Useful Higher Order Functions
Map
(1 to 10).map(0.1 * _)

foreach

(1 to 10).map("*" * _).foreach(println _)

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 22

 Useful Higher Order Functions (Cont’d)
filter
(1 to 15).filter(_ % 2 == 0)

reduceLeft

(1 to 5).reduceLeft(_ * _)

Split, sortWith

“Twinkle Twinkle Little Star".split(" ").sortWith(_.length < _.length)

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 23

 Anonymous Functions

ᗍ Passing functions as parameters leads to the creation of many small functions

ᗍ Sometimes it is tedious to have to define (and name) these functions using def
ᗍ Compare to strings: We do not need to define a string using def

Instead of
def str = ”skillspeed”; println(str)

We can directly write

println(”skillspeed”)

ᗍ Because strings exist as literals. Analogously we would like function literals, which let us write a function without
giving it a name
ᗍ These are called anonymous functions

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 24

 Anonymous Function Syntax

ᗍ Example: A function that raises its argument to a cube:

(x: Int) => x * x * x

ᗍ Here, (x: Int) is the parameter of the function, and x * x * x is it’s body
ᗍ The type of the parameter can be omitted if it can be inferred by the compiler from the context
ᗍ If there are several parameters, they are separated by commas:

(x: Int, y: Int) => x + y

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 25

 Simplified Anonymous Functions
Anonymous Function can be made even more simplified in the following manner

//Explicit type declaration

val call1 = doWithOneAndTwo((x: Int, y: Int) => x + y)

//The compiler expects 2 ints so x and y types are inferred

val call2 = doWithOneAndTwo((x, y) => x + y)

//Even more concise syntax

val call3 = doWithOneAndTwo(_ + _)

This simplification of syntax is also more commonly and formally known as Syntactic Sugar

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 26

 Check your Understanding – 2

Functions can’t be passed as arguments in Scala

a. True
b. False

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 27

 Check your Understanding – Solution

Functions can’t be passed as arguments in Scala

a. True
b. False

False

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 28

 Check your Understanding – 3

Anonymous functions can access the out of bound variables

a. True
b. False

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 29

 Check your Understanding – Solution

Anonymous functions can access the out of bound variables

a. True
b. False

True

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 30

 Parameter Inference
ᗍ If one function is passed as parameter to another function, Scala helps us in deducing types wherever possible:
Example:

ᗍ If a function has only one parameter, then the “()” can be omitted, hence now it can be expressed as:
value(x => 4 * x)

ᗍ If the parameter is used only ONCE on right side of =>, then it can be replaced with underscore “_” . So now

value(4 * _)

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 31

 Closures
ᗍ In Scala, the functions can be defined anywhere, in a package or class or inside another function or method

ᗍ We can access the variables of enclosing scope from the function

ᗍ When defining a function, at runtime we get an object. Each function call is actually an object. When we define a
function that uses variables from its outer scope the object that we get is a closure

ᗍ Example:

ᗍ This is called Closure. So, a Closure comprises of code along with the definition of non-local variables used by the
code

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 32

 Currying

ᗍ Currying is the technique of transforming a function that takes multiple arguments into a function that takes a
single argument
Example:
//before currying

def add(a:Int, b:Int) = a + b

add(1, 2) // 3
add(7, 3) // 10

//After Currying

def add(a:Int) = (b:Int) => a + b

add(1)(2) // 3
add(7)(3) // 10

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 33

 File Processing: A Quick Look

ᗍ To read all lines from a file, use getLine method

ᗍ The result is an iterator, which then can be used to process line one at a time
ᗍ You can use fromFile also to read all lines in a file. mkString converts a collection into a flat String by each
element's to String method

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 34

 Implicit Function Parameter

ᗍ A method with implicit parameters can be applied to arguments just like a normal method
ᗍ In this case the implicit label has no effect
ᗍ However, if such a method misses arguments for its implicit parameters, such arguments will be automatically
provided

Call by Value:
ᗍ Typically, parameters to functions are by-value parameters; that is, the value of the parameter is determined before
it is passed to the function
ᗍ In most circumstances, this is the behaviour we want and expect

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 35

 Example

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 36

 Call by Name
If we need to write a function that accepts as a parameter an expression that we don't want evaluated until it's called
within our function. Scala offers call-by-name parameters in such cases

The output is different:

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 37

 Check your Understanding – 4

Abstract methods of a trait must be overridden in subclass

a. True
b. False

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 38

 Check your Understanding – Solution

Abstract methods of a trait must be overridden in subclass

a. True
b. False

False

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 39

 Check your Understanding – 5

Closures:

a. Can’t access out of scope variable

b. Can access out of scope variable
c. Essentially are anonymous functions

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 40

 Check your Understanding – Solution

Closures:

a. Can’t access out of scope variable

b. Can access out of scope variable
c. Essentially are anonymous functions

Can access out of scope variable and Essentially are anonymous functions

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 41

 Questions

© 2015 BlueCamphor Technologies (P) Ltd. www.skillspeed.com Slide 42