CPPDS Slide.pdf

Certified Python Programmer For
Data Science (CPPDS)

Course Outlines
Day 1
• Introduction to Programming
• Python Basics
Day 2
• Jupyter Notebook
• Functions in Python
• Pythonic Programming
• Modules and Packages
• Object Oriented Programming

Course Outlines
Day 3
• OS Module
• Working with Files
• Working with JSON Data
• Web Scraping (BeautifulSoup)
Day 4:
• Introduction to Matrix Processing (NumPy)
• Data Analysis (Pandas Module)
• Data Visualization (Matplotlib)

Course Outlines
Day 5
• Introduction to Applied Machine Learning
• Capstone Project
• Course Exam

Day 1
• Python Basics

Module 1
Introduction to Programming

Outlines
 What is Algorithm?
 What is Programming?
 The Natural Language of the Computer.
 Programming Languages Levels.
 Translators
 Which Programming Language and Why?

What is Algorithm?
 Algorithm is a set of processes defined and completed for the
completion of a work or to solve a problem and obtain a logical
result.

Example Problem
 Given a list of positive numbers, return the largest number on the
list.

Example Analysis
 Inputs: a list (L) of positive numbers.
 Outputs: a number(n), which will be the largest number of the list.

Example Algorithm
 Set max to 0.
 For each number x in the list L, compare it to max. If x is larger,
set max to x.
 Max is now set to the largest number in the list.

What is Programming?
 Programming is coding, modeling, simulating or presenting the
solution to a problem, by representing facts, data or information
using pre-defined rules and semantics, on a computer or any other
device for automation.

What is a Program?
 A program is a set of commands written in a specific language to
be arranged in a logical order to solve a specific problem.

Programming Language
 Programming language is the means through which the computer
can receive the instructions of the algorithms required to solve the
applied problems..

The Natural Language of the Computer
 Machines have their natural language like humans do.
 Computers do not understand the human language.
 The natural language of computers is the binary code(1 and 0).
 These represent two states: on (1) and off (0).
 That is the natural language of electronic equipment.
 It would be hectic for us as humans to communicate with the
computer in binary.

Programming Languages Levels
 Low level programming languages.
 High level programming languages.

Low Level Programming Languages
 Represent the instructions in a manner that resembles more
closely the way the computer actually works.
 For example:
Machine language (whose instructions are written in binary code
(0 or 1))

High Level Programming Languages
 High level languages do resemble more the way the human mind
works.
 It needs to translator.
 For example
1. C++
2. Python
3. Java
4. Basic
….

Compare…
 Each type is a good fit depending on the particular case.
 Whenever speed and resource consumption are important, low
level languages can provide an advantage since they cause much
less "translation" and are less generic, thus causing less load on
the computer.
 High level languages have much more abstraction and thus are
better suited for tasks where maintenance and complex design is
required.

Translator
 Source code refers to a program was written in a specific
programming language that a person understands.
 Object code refers to a program written in machine language..
 Translator has the responsibility of converting your source code to
the machine language.

Translators
 Translators can be any of:
Interpreters
Compilers

Which Programming Language and Why?
 Data science mainly can be summarized as
following

Which Programming Language and Why?
 During these phases you need a programming language.
 Python is the programming language number 1 in the world for
data scientists.

Summary
 What is Algorithm?
 What is Programming?
 The Natural Language of the Computer.
 Programming Languages Levels.
 Translators
 Which Programming Language and Why?

Outlines
 What is Python
 Python Features
 Some of Python Libraries
 Python Identifiers
 Comment in Python
 Variables
 Built in Functions
 Python Lists
 Copy in Python

Outlines
 Python Tuples
 Python Dictionaries
 Python Sets
 Python Strings
 Type Conversion
 Python Operators
 Decision Making
 Python Loops

What is Python?
 Python is a high-level, interpreted, object-oriented and scripting
language.
 Python is designed to be highly readable where it uses English
keywords frequently

History of Python
 Python was developed by Guido van Rossum in the late eighties
and early nineties at the National Research Institute for
Mathematics and Computer Science in the Netherlands.
 Python is derived from many other languages, including C, C++
and others

Python Features
 Easy-to-Learn
 Easy-to-Read
 Memory Management
 Dynamic Data Types
 A Broad Standard Library
 Portable
 Databases
 GUI Programming

First Python Script
 Open Jupyter Notebook
print("Hello World")

Python Identifiers
 A Python identifier is a name used to identify a variable, function, class,
module or other object.
 An identifier starts with a letter A to Z or a to z or an underscore (_) followed
by zero or more letters, underscores and digits (0 to 9).
 Python does not allow punctuation characters such as @, $, and % within
identifiers
 Python is a case sensitive programming language. Thus, Manpower and
manpower are different

Reserved Words
 The following list shows the Python keywords.

Lines and Indentation
 Python provides no braces to indicate blocks of code
 Blocks of code are denoted by line indentation, which is rigidly enforced.
 The number of spaces in the indentation is variable, but all statements within
the block must be indented the same amount

Multi-Line Statements
 Statements in Python typically end with a new line
 Python does, however, allow the use of the line continuation
character () to denote that the line should continue

Multi-Line Statements
 Statements contained within the [], {}, or () brackets do not need
to use the line continuation character

Multiple Statements on a Single Line
 The semicolon ( ; ) allows multiple statements on the single line

Comments in Python
 A hash sign (#) begins a comment
 You can use triple quotes """ """

Variables
 Reserved memory locations to store values.
 Based on the data type of a variable, the interpreter allocates the required
memory space.
 Python variables do not need explicit declaration to reserve memory space.
 The equal sign (=) is used to assign values to variables.

Variables - Assignment
 Example
counter = 100 # An integer number
miles = 1000.0 # A floating point number
name = "John" # A string
a = b = c = 1
a, b, c = 1 ,2 , "john"

Variables - Standard Data Structures
Python supports 6 major data structures:
 Number (int, float, complex)
 String
 List
 Tuple
 Dictionary
 Set

Built-in Functions
 min(x1, x2,...): the smallest of its arguments.
 max(x1, x2,...): the largest of its arguments.
 round(x): x rounded to n digits from the decimal point.

Mathematical Functions (math)
 log(x): the natural logarithm of x, for x> 0 .
 pow(x, y): the value of x**y.
 exp(x): the exponential of x .

Mathematical Functions (math)
 fabs(x): the absolute value of x: the (positive) distance
between x and zero.
 sqrt(x): the square root of x for x > 0 .

Trigonometric Functions (math)
 sin(x): returns the sine of x radians.
 cos(x): returns the cosine of x radians.
 tan(x): returns the tangent of x radians.

Trigonometric Functions (math)
 degrees(x): converts angle x from radians to degrees.
 radians(x): converts angle x from degrees to radians.

Mathematical Constants (math)
 pi: the mathematical constant pi.
 e: the mathematical constant e.

Random Number Functions (random)
 random: arandom float r, such that 0 is less than or equal to r and
r is less than 1.
 randint: a random integer in the range x,y

Challenge
 Compute the square root of 9, 16, 100
 Convert pi into degree
 Generate a random integer number between 4 and 7

Summary
 Python features
 Identifiers
 Comments
 Variables
 Built in function

Python Lists
 Lists are the most versatile of Python's compound data types.
 A list contains items separated by commas and enclosed within
square brackets ([ ]).
 A list is an ordered collection of mutable items which can be of
any type.
 The first index is zero, the second index is one, and so forth.

Accessing Values in Lists
 Example:
list1 = ['physics', 'chemistry', 1997, 2000]
list2 = [1, 2, 3, 4, 5, 6, 7 ]
print ("list1[0]: ", list1[0])
print ("list2[1:5]: ", list2[1:5])

Updating Lists
 Example:
list = ['physics', 'chemistry', 1997, 2000]
print ("Value available at index 2 : ")
print (list[2])
list[2] = 2001
print ("New value available at index 2 : ")
print (list[2])

Delete List Elements
 Example:
list1 = ['physics', 'chemistry', 1997, 2000]
print (list1)
del list1[2]
print ("After deleting value at index 2 : ")
print (list1)

Built-in List Functions & Methods
 len : returns the number of items in the list.
 max : returns item from the list with max value.
 min : returns item from the list with min value.
 list.append: appends object obj to list.

Built-in List Functions & Methods
 list.count: returns count of how many times obj occurs in list.
 list.index: returns the lowest index in list that obj appears.
 list.remove: removes object obj from list.
 list.reverse: reverses objects of list in place.

Copy List
 Example :
a = [1, 2, 3, 4, 5]
b = a
c = a.copy()
print(a)
a[0]=7
print(a)
print(b)
print(c)

Copy in Python
 Assignment statements in Python do not copy objects, they create
bindings between a target and an object.
 For collections that are mutable or contain mutable items, a copy
is sometimes needed so one can change one copy without
changing the other.

Shallow Copy
 In case of shallow copy, a reference of object is copied in other
object. It means that any changes made to a copy of object do
reflect in the original object.

Deep Copy
 In case of deep copy, a copy of object is copied in other object. It
means that any changes made to a copy of object do not reflect in
the original object.

Challenge
a = [4, 7, 2, 9, 3, 2]
 Find the min, max, sum
 How many 2s?

Challenge
letters =['A','C','C','A','T','T','G','A','C','A']
 Analyze the letters
 Count A in letters
 Find out which location is first T
 Manipulate the letters
 Copy (DEEP) from original letters to letters2
 Remove G from letters2
 Insert A in index 3 in letters2
 Reverse the order of new letters

Lists Summary
Bracket Mutable Ordered
[ ] True True

Python Tuples
 A tuple consists of a number of values separated by commas and
enclosed within parentheses ().
 A tuple is an ordered collection of immutable items which can be
of any type.
 Tuples can be thought of as read-only lists
tu = ( 'abcd', 786, 2.23, 'john', 70.2 )

Accessing Values in Tuples
 Example:
tup1 = ('physics', 'chemistry', 1997, 2000)
tup2 = (1, 2, 3, 4, 5, 6, 7 )
print ("tup1[0]: ", tup1[0])
print ("tup2[1:5]: ", tup2[1:5])

Updating Tuples
 Example:
tup = (12, 34.56)
What will happen ?
tup[0] = 100

Built-in Tuple Functions
 len: gives the total length of the tuple.
 max: returns item from the tuple with max value.
 min: returns item from the tuple with min value.
 tuple: converts a list into tuple.

Challenge
With a given tuple (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
 Write a program to print the first half values in one line and the
last half values in one line.

Tuples Summary
( ) False True

Python Dictionaries
 A dictionary is a list of items.
 Each item is associated with a key and value.
 Each key is separated from its value by a colon (:)
 The items are separated by commas
 The whole thing is enclosed in curly braces

Create a Dictionary
 Example:
d1 = dict(a=1, b=2)
d2 = {'a': 1, 'b': -1}
d3 = dict(zip(['a', 'b'], [1, 2]))
d4 = dict([('a', 1), ('b', 3)])
d5 = dict({'b': 2, 'a': 1})

Accessing Values in Dictionary
 Example:
dict1 = {'Name': 'Zara', 'Age': 7, 'Class':
'First'}
print ("dict1['Name']: ", dict1['Name'])
print ("dict1['Age']: ", dict1['Age'])

Updating Dictionary
 Example:
dict1 = {'Name': 'Zara', 'Age': 7, 'Class': 'First'}
dict1['Age'] = 8 # update existing entry
dict1['School'] = "DPS School" # Add new entry
print ("dict1['Age']: ", dict1['Age'])
print ("dict1['School']: ", dict1['School'])

Delete Dictionary Elements
 Example:
dict1 = {'Name': 'Zara', 'Age': 7, 'Class': 'First'}
del dict1['Name'] # remove entry with key 'Name'
dict1.clear() # remove all entries in dict
del dict1 # delete entire dictionary
print ("dict['Age']: ", dict['Age'])
print ("dict['School']: ", dict['School'])

Properties of Dictionary Keys
 There are two important points to remember about dictionary
items :
◦ More than one entry per key not allowed
◦ Keys must be immutable. For example: tuple – string

Built-in Dictionary Functions & Methods
 len: returns the number of items in the dictionary.
 dict.copy: returns a deep copy of dictionary dict.
 dict.items: returns a list of dict's (key, value) tuple pairs.
 dict.keys: returns a list of dictionary dict's keys.
 dict.values: returns a list of dictionary dict's values.

Challenge
thisdict = dict('apple'="green", 'banana'="yellow", 'cherry'="red")
 Remove apple
 Add durian
 How many elements?
 Create a shallow copy?
 Create a deep copy?

Dictionaries Summary
{ } True False

Python Sets
 A set object is an unordered collection of distinct objects
a = {1, 2, 3, 4, 5}
a = set([2, 3, 4, 5, 6])
1 in a
3 not in a

Built-in Set Functions
 len: gives the total length of the tuple.
 a.issubset(b): returns True if a is a subset of b (a <= b).
 a.issuperset(b): returns True if b is a subset of a (a>=b).

Built-in Set Functions
 a.union(b): returns the union of two sets (a | b).
 a.intersection(b): returns the intersection of two sets (a & b).
 a.add(elem): add a new element to the set

Challenge
engineers = set(['John', 'Jane', 'Jack', 'Janice'])
programmers = set(['Jack', 'Sam', 'Susan', 'Janice'])
managers = set(['Jane', 'Jack', 'Susan', 'Zack'])
 Who are employees?
 Who are engineers and managers?

Sets Summary
{ } True False

Summary
 List : [ Bracket ]
 Tuple: ( Parenthesis )
 Dictionary/Set: { Braces }

Python Strings
 Strings in Python are identified as a contiguous set of characters
represented in the quotation marks.
 Python allows for either pairs of single, double, or triple quotes.

String Special Operators
 If a = "Hello" , b = "Python"

Accessing Values in Strings
 Example:
var1 = 'Hello World!'
var2 = "Python Programming"
print ("var1[0]: ", var1[0])
print ("var2[1:5]: ", var2[1:5])

Built-in String Functions
 len: returns the length of the string.
 split: splits string according to delimiter str (space if not
provided).
 replace: replaces all occurrences of substring with another
substring.
 endswith: returns ture if string end with given suffix

 isspace: returns true if string contains only whitespace.
 isdigit: returns true if string contains only digits
 capitalize: capitalizes first letter of string.

 count: counts how many times substring occurs in string.
 lower: converts all letters in the string to lowercase.
 find: determines if substring occurs in string.

String Formatting
 There are three ways:
name = "John"
print("Hello, %s!" % name)
print("Hello, {}!" .format(name))
print(f "Hello, {name}!")

Basics
 %s - String (or any object with a string representation, like
numbers)
 %d – Integers
 %f - Floating point numbers
 %.<number of digits>f - Floating point numbers with a fixed
amount of digits to the right of the dot.

Old Style
 Example:
template = "%s is %d years old."
name = "John"
age = 23
print(template % (name, age)) # This prints out "John is
23 years old."

Old Style
 Any object which is not a string can be formatted using the %s
operator
template = "A list: %s"
mylist = [1,2,3]
print(template % mylist)

New Style
 Example:
template = "{} + {} = {}"
a = 3.4567
b = 4
c = a +b
print(template.format (a, b, c))
template = "{1} + {0} = {2}"
print(template.format(a, b, c))

New Style
 Example:
template = "{:0.2f} + {:0.2f} = {:0.2f}"
template = "{0:0.2f} + {1:0.2f} = {2:0.2f}"

Challenge
a = 'abc@def.com'
 Print out the user name and domain name

Challenge
 You have to write a format string which prints out the following
sentence:
Hello John Doe. Your current balance is $53.44
 Given this data tuple
data = ("John", "Doe", 53.44)

Challenge
country = "France"
capital = "paris"
 Print out the following sentence using {} method
 The capital of FRANCE is Paris

Strings Summay
'' "" """""" False True

Type Conversion
 type(a) - show the variable data
 list(a) - convert to a list
 set(a) - convert to a set
 tuple(a) - convert to a tuple
 int(a) - convert to an integer
 float(a) - convert to a float
 str(a) - convert to a string

Data Structures Summary
List [ ] True True
Tuple ( ) False True
Set { } True False
Dictionary { } True False
String '' "" """""" False True

Python Operators
 Operators are the constructs which can manipulate the value of
operands
 Consider the expression 4 + 5 = 9.
 Here, 4 and 5 are called operands and + is called operator.

Types of Operator
 Python language supports the following types of operators:
Arithmetic Operators
Comparison (Relational) Operators
Assignment Operators
Logical Operators
Membership Operators
Identity Operators

Python Arithmetic Operators
 Assume variable a holds 10 and variable b holds 20, then

Python Comparison Operators

Python Assignment Operators

Python Logical Operators
 Assume variable a holds T and variable b holds F then

Python Membership Operators
 Python's membership operators test for membership in a
sequence, such as strings, lists, or tuples

Membership Operators -Example
 Example:
a = 10
b = 20
lst = [1, 2, 3, 4, 5 ]
print (a in lst )
print (b not in lst)

Python Identity Operators
 Identity operators compare the memory locations of two objects

Identity Operators - Example
a = 20
b = 20
 print (a is b) why ?

Python Operators Precedence
 The following table lists all operators from highest precedence to
lowest.

Operators Precedence - Example
 Example:
a = 20; b = 10; c = 15; d = 5
e = (a + b) * c / d #( 30 * 15 ) / 5
print (e)
e = ((a + b) * c) / d # (30 * 15 ) / 5
print (e)
e = (a + b) * (c / d)
print (e)

Challenge
a = 20
b = 10
c = 15
d = 5
 What is the answer of
e = a + (b * c) / d ?

Summary
String
Operators
Type Conversion

Decision Making – Control Structures
 Decision making is about taking actions according to the
conditions.
 Decision structures evaluate multiple expressions which produce
TRUE or FALSE as outcome.

If Syntax
if condition:
statement(s)

If - else Syntax
if condition:
statement(s)
else:
statement(s)

If - else Syntax
 Example:
a=5
b=3
if a < b:
print("a is smaller than b")
else:
print("a is larger than b")

If - elif- else Syntax
if expression1:
statement(s)
elif expression2:
statement(s)
.
.
.
elif expressionN:
statement(s)
else: # with else or without else
statement(s)

If - elif- else Example
 Example:
a=5
b=3
if (a < b):
print("a is smaller than b")
elif (a > b):
print("a is larger than b")
else:
print("a is same as b")

The Ternary Operator
 Example:
order_total = 200
discount = 25 if order_total > 100 else 0

Challenge
grade = input('What is your grade :' )
 if grade is A, then print "Excellent!"
 if grade is B, then print "Well done!"
 if grade is C, then print "Work harder "
 otherwise, print "I don't know your grade"

Python Loops
 In general, statements are executed sequentially
 There may be a situation when you need to execute a block of
code several number of times.
 A loop statement allows us to execute a statement or group of
statements multiple times.

While Loop Syntax
while condition:
statement(s)

While Loop - Example
 Example:
a = 1
while (a < 10):
print(a)
a = a + 1

For Loop Syntax
for variable in sequence:
statements(s)

For Loop - Example
 Example:
for x in [1, 2, 3, 4, 5]:
print("x = ",x)
fruits = ['Durian', 'Banana','Orange','Apple']
for today_fruit in fruits:
print ("Today food is ", today_fruit)

Range
 Range generates a sequence of numbers
range(start, stop, step)
a = range(1,5)
b = range(10,-1,-1)

Range - Example
 Example:
a = []
for i in range(1,11):
a.append(i*i)

Enumerate
 Example:
person = ['Alfred', 'Ally', 'Belinda']
height = [170, 160, 155]
for i, name in enumerate(person):
h = height[i]
print("The height of {} is {}cm".format(name, h))

Zip
 Instead of using nested loops
a = [1, 2, 3, 4, 5, 6]
b = ['a', 'b', 'c', 'd', 'e',' f']
c = zip(a, b)
for i, j in c:
print(i, j)

Loop Control Statements
 Loop control statements change execution from its normal
sequence

Break - Example
 Example:
for letter in 'Python':
if letter == 'h':
break
print ('Current Letter :', letter)

Continue - Example
 Example:
for letter in 'Python':
if letter == 'h':
continue
print ('Current Letter :', letter)

Loop with Else
• Python allows else clauses after while and for loops
• If the loop ends normally, then the else clause is executed.
• If the loop is interrupted by a break statement, the else clause is
not executed

Else in Loop
 Example:
x = [1, 2, 3, 4]
for i in x:
if i % 5 == 0:
break
print(i)
else:
print("no break")

Challenge
 Write a program which will find all such numbers which are
divisible by 7 but are not a multiple of 5, between 2000 and 3200
(both included)
 Write a program which can compute the factorial of a given
number.

Summary
 What is Python
 Python Features
 Some of Python Libraries
 Python Identifiers
 Comment in Python
 Variables
 Built in Functions
 Python Lists
 Copy in Python

Summary
 Python Tuples
 Python Dictionaries
 Python Sets
 Python Strings
 Type Conversion
 Python Operators
 Decision Making
 Python Loops

Day 2
 Jupyter Notebook
 Functions in Python
 Pythonic Programming
 Modules and Packages
 Object Oriented Programming

Outlines
 Running Jupyter Notebook
 Components
 Jupyter Notebook Interface
 Modes
 CheckPoints
 Markdown Language

Jupyter Notebook
 Jupyter Notebook is a web application that allows you to create
and share documents that contain:
• Python code
• Visualization graphs
• Explanatory text (written in markdown syntax)

How to Install Jupyter Notebook
 You can install Jupyter Notebook in two different ways:
• Installing Jupyter Notebook by using the Python's package
manager pip
• Installing Jupyter Notebook by installing the Anaconda
distribution (https://www.anaconda.com/download)

Running Jupyter Notebook
 Using the following command (in the command prompt or in the
Anaconda prompt):
jupyter notebook

Components
 The Jupyter Notebook combines three components:
• The notebook web application (GUI)
• Kernels
• Notebook documents

The Notebook Web Application
 An interactive web application for writing and running code
interactively and authoring notebook documents.

Kernels
 For each notebook document that a user opens, the web
application starts a kernel that runs the code for that notebook.
 Each kernel is capable of running code in a single programming
language

Notebook Documents
 Notebook documents contain the inputs and outputs of an
interactive session as well as narrative text that accompanies the
code but is not meant for execution.
 Rich output generated by running code, including HTML, images,
video, and plots, is embedded in the notebook, which makes it a
complete and self-contained record of a computation.

Jupyter Notebook Interface
 The web server is started and the Jupyter Notebook application is
opened in your default browser automatically.

The User Interface
 As you can see the user interface of Jupyter Notebook is split up
into three sections (tabs):
• Files
• Running
• Clusters

Creating a New Notebook
 Just use the New dropdown menu and you'll see the following
options:

Creating a New Notebook
 Select option Python 3 to open a new Jupyter Notebook for
Python.
 The notebook is created:

Naming Jupyter Notebook
 By giving it a name the notebook will also be saved as a file of
the same name with extension .ipynb.

Working With the Notebook
 The notebook itself consists of cells. A first empty cell is already
available after having created the new notebook:
 This cell is of type "Code" and you can start typing in Python
code directly.

First Code
 Executing code in this cell can be done by either clicking on the
run cell button or hitting Shift + Return keys:

Change the Cell Type
 You can change the cell type from Code to Markdown to include
explanatory text in your notebook.
 To change the type you can use the dropdown input control:

Change the Cell Type
 Once switched the type to Markdown you can start typing in
markdown code:

Modes
 If a cell is active two modes distinguished:
edit mode
command mode

Edit Mode
 The edit mode is entered if you click into the code area of that
cell. This mode is indicated by a green border on the left side of
the cell:
 If you'd like to leave edit mode and return to command mode
again you just need to hit ESC.

Command Mode
 If you just click in one cell the cell is opened in command mode
which is indicated by a blue border on the left:

Overview Functions
 To get an overview of functions which are available in command
and in edit mode you can open up the overview of key shortcuts
by using menu entry
Help → Keyboard Shortcuts

Checkpoints
 To create a new checkpoint for your notebook select menu item
Save and Checkpoint from the File menu.
 The checkpoint is created and the notebook file is saved.
 If you want to go back to that checkpoint at a later point in time
you need to select the corresponding checkpoint entry from menu
File → Revert to Checkpoint.

Exporting the Notebook
 Jupyter Notebook gives you several options to export your
notebook.
 Those options can be found in menu File → Download as:

Markdown Language
 You can document the computational process in a literate way,
alternating descriptive text with code, using rich text.
 In Python this is accomplished by marking up text with the
Markdown language.
 The corresponding cells are called Markdown cells.

Markdown Language
 The Markdown Language provides a simple way to perform this
text markup, that is, to specify which parts of the text should be
emphasized (italics), bold, form lists, etc.

Markdown Language
 If you want to provide structure for your document, you can use
markdown headings.
 Markdown headings consist of 1 to 6 hash # signs followed by a
space and the title of your section.
 You can emphasis some parts by using:
• Bold: __string__ or **string**
• Italic: _string_ or *string*

Markdown Language
• For more details refer to :
https://medium.com/ibm-data-science-experience/markdown-for-
jupyter-notebooks-cheatsheet-386c05aeebed

Summary
 Running Jupyter Notebook
 Components
 Jupyter Notebook Interface
 Modes
 CheckPoints
 Markdown Language

Outlines
 What is Function?
 Passing Named arguments
 Lambda Function
 Map Function

What is Function?
 A block of code that perform a specific task:
Calculating a mathematical expression
Evaluation the outcome of a series of statements
Printing results in a specific format

Function Syntax
def func_name(input list ):
code
return output expression
def f(x):
return x*x
print (f(3))

Return Multiple Values
 Example:
def f(x):
return (x*x, 2+x)
print (f(3))

Passing Named Arguments
 Example:
def add(x, y):
z = x + y
return z
print (add(3, 4))
def rect(a=3, b=5):
return a*b
print (rect())
print(rect(b=8))
print(rect(b=6, a=3))

Variable Arguments
 Example:
def sum1(a,*b):
s= a
for i in b:
s = s +i
return s

Challenge
 Write a Python function that checks whether a passed string is
palindrome or not
 Give a sequence of variable number of elements, create a function
to determine the minimum value
 Write a Python function that takes a number as a parameter and
check the number is prime or not

Lambda Function
 Syntax:
name = lambda input list: output expression
 equivalent to :
def name(arg1, arg2,....):
return expression.

Lambda Function - Example
 Example:
f = lambda x: x*x
equivalent to
def f(x):
return x*x
Exp
G = lambda x, y: x*y

When to Use Def or Lambda
 When you don't need the function name, use lambda, eg passing a
function to another function
 When you need to use the function name to re-use again, use def

Challenge
 Create a lambda function to sum the elements of two lists
list_a = [1, 2, 3]
list_b = [10, 20, 30]

Map Function Syntax
 map(function, seq)
 map is very useful when you have to apply the same function to
one or more collections of objects
a = [3, 4, 5, 6, 7]
f = lambda x:x**2
print(list(map(f,a)))

Map - Example
 Example:
a = [0,1,2,3]
b = [2,4,6,8]
print(list(map(lambda x,y:10*x+y, a,b)))

Summary
 What is Function?
 Passing Named arguments
 Lambda function
 Map function

Outlines
 List Comprehension
 Generator Expressions
 Exceptions Handling

List Comprehension
 Comprehensions are concise, syntactic constructs that allow
sequences to be built from other sequences.
 Python 2.0 introduced list comprehensions and Python 3.0 comes
with a dictionary and set comprehensions.
 It creates a new list by applying an expression to each element in
an iterable variable.

Syntax
 [ <expression> for <element> in <iterable> ]
 [ <expression> for <element> in <iterable>
if <condition> ]
 [ <expression> if <condition> else <statement>
for <element> in <iterable> ] (Careful here)

Syntax - Examples
 Example:
squares = [x * x for x in (1,2,3,4)]
print (squares)
Equivalent to:
squares = []
for x in (1,2, 3, 4):
squares.append(x * x)

Syntax - Example
 With if:
 With if else:
s1 = [s.upper() for s in 'hello' if s == 'h']
s2 = [s if s in 'arial' else '*' for s in 'apple']

Challenge
 Create a list of squares of the numbers from 1 to 20 where you
have to include only the numbers that are not divisible by 3.
 Use comprehension method to extract the first letter of each word
in the sentence below:
Sentence : 'Today is a great day'
The output should be
["T", "I", "a", "g", "d"]

Challenge
 Use list comprehension to return a list of odd numbers in the
range 0 - 10

Generator Expressions
 An iterator is an object that can be iterated (looped) upon
 For example:
 Generator Expressions allow us to create iterators in a more
simple fashion.
 Generators compute the value only when you ask for it (lazy
evaluation)
for i in range(50):
print(i)

Create Generator Object
 Generator object can be created by:
Generator Expression
Generator Function

Generator Expressions Syntax
 ( <expression> for <element> in <iterable> )
 (<expression> for <element> in <iterable> if <condition> )
 (<expression> if <condition> else <statement>
for <element> in <iterable> ) (Careful here)

Generator Expression
 Example:
(n*n for n in range(10) )
(n*n for n in range(10) if n % 2 == 0)
sum(i*i for i in range(10) )
for i in (n*n for n in range(10)):
print(i)

Next
 Example:
a = (n*n for n in range(10))
next(a)

Generator Function
 Use Yield instead of Return
 Return one value at one time
def range2(n):
for i in range(n):
if i%2 == 0:
yield i
for i in range2(10):
print(i)

Challenges
Name_list = ['ally', 'jane', 'belinda']
 Create an upper case name list generator using generator
expression

Exceptions Handling
 An exception is an event, which occurs during the execution of a
program that disrupts the normal flow of the program's
instructions.
 In general, when a Python script encounters a situation that it
cannot cope with, it raises an exception
 When a Python script raises an exception, it must either handle
the exception immediately otherwise it terminates and quits.

Standard Exceptions
 Exception: Base class for all exceptions.
 StopIteration: Raised when the next() method of an iterator does not
point to any object.
 ZeroDivisonError: Raised when division by zero takes place for all
numeric types.
 ImportError: Raised when an import statement fails.
 IndexError: Raised when an index is not found in a sequence.
 Etc …

The Except Clause with no Exceptions
try:
You do your operations here
......................
except:
If there is any exception, then execute this block.
......................
else: # Optional
If there is no exception then execute this block.

The Except Clause with Specific Exceptions
try:
You do your operations here
......................
except ExceptionI:
If there is ExceptionI, then execute this block.
.
.
.
except ExceptionN:
If there is ExceptionII, then execute this block.
......................
else: # Optional
If there is no exception then execute this block.

Example

try:
fh = open("testfile", "w")
fh.write("This is my test file for exception handling!!")
except IOError:
print ("Error: can't find file or read data")
else:
print ("Written content in the file successfully")
fh.close()

Example

try:
a = 2/4
except ZeroDivisionError:
print ("Zero division error")
else:
print ("Correct ....")

Challenge
a = [1, 2, 3]
b = a[8]
 This code will produce an exception, catch it?

Challenge
 Read the following code
elements = (x for x in range(5))
for elem in elements:
print(elem)
 Replace for by while and do the required corrections

Summary
 List Comprehension
 Generator Expressions
 Exceptions Handling

Outlines
 Modules
 Documentation
 Packages
 Namespace

Modules
 In programming, a module is a piece of software that has a
specific functionality
 For example, when building a ping pong game:
• One module would be responsible for the game logic
• Another module would be responsible for drawing the game on
the screen

Modules
 Modules in Python are simply Python files with a .py extension.
 mygame/game.py mygame/draw.py
 The name of the module will be the name of the file.
 A Python module can have a set of functions, classes or variables
defined and implemented

Import Modules
 Import everything:
 Import with abbreviation
import math
math.sin(0.5)
import math as m
m.sin(0.5)

Import Names from a Module
 Import everything
 Import just a few selected names
from math import *
sin(0.5)
from math import sin,pi
sin(0.5)

Module Initialization
 The first time a module is loaded into a running Python script, it
is initialized by executing the code in the module once.
 If another module in your code imports the same module again, it
will not be loaded twice but once only.
 This is useful to know, because this means that you can rely on
this behaviour for initializing objects.

Exploring Built-in Modules
 Two very important functions come in handy when exploring
modules in Python:
dir()
help()
import requests
dir(requests)
help(requests.urlopen)

__name__ variable or main guard
def main():
…..
if __name__ == '__main__':
main()
main() will be executed only when we run the module from the
module itself not when we import the module from another module

Documentation
 Use triple quotes for documentation in case of functions, classes,
modules ....
 This documentation will appear in the output of help function
def func():
"""
The input ….
This function will …
"""

Packages
 Packages are directories which contain multiple packages and
modules themselves
 Each package in Python is a directory which MUST contain a
special file called __init__.py

Example
 Create a directory called foo
 Create a module inside that package called bar
 Add the __init__.py file inside the foo directory
 import foo.bar (Use foo prefix)
 Or: from foo import bar (No need for foo prefix)

Challenge
 Create a module "area_calculation"
 This module should have a function to compute the area of the rectangle
"rect_area" and a function to compute the area of the circle "circle_area"
 Create a module with the name "area_test"
 Import "area_calculation" module and use the available functions
 Move the "area_calculation" module into a package of the name "area"
 Do the required corrections.

What is Namespace?
 Is a collection of names.
 Different namespaces can co-exist at a given time but are
completely isolated.
 A namespace containing all the built-in names is created when we
start the Python interpreter and exists as long we don't exit.

Summary
 Modules
 Documentations
 Packages
 Namespace

Module 7
Object Oriented Programming

Outlines
 What is OOP?
 Important Definitions
 Attributes Types
 Methods Types
 Relations between Classes

What is OOP?
 Object-oriented programming (OOP) focuses on creating
reusable patterns of code.
 Abstraction is one of the core concepts of OOP.
 Abstract (for anything what are the attributes and actions).
 It models applications as a group of related objects that interact
with each other.

Important Definitions
 Class: it is like a blueprint or a prototype which has: attributes and
methods.
 Attributes: are the properties that specify the information associated
with each class object.
 Methods: defines object behaviors that depends on its attributes.
 Object: is an instance of a class, which wrapping pieces of data, and
behavior related to that data.

Challenge
 Create a Counter class with a property of count.
 This class has 3 functions:
Increasing the count by 1
Increasing the count by n
Reset the count to 0
The initial value of count is 0

Attributes Types and Methods Types
 Attributes:
Instance Attributes
Class Attributes
 Methods:
Instance Methods
Static Methods
Class Methods

Instance Attributes
 Python variable belonging to one, and only one, object.
 This variable is only accessible in the scope of this object.
 It is defined inside the class constructor.

Instance Attributes

# Instance attributes
class Shark:
def __init__(self, name):
self.name = name
def swim(self):
# Reference the name
print(self.name + " is swimming.")
def be_awesome(self):
# Reference the name
print(self.name + " is being awesome.")
sammy = Shark("Sammy")
sammy.swim()
sammy.be_awesome()

Class Attributes
 Python variable that belongs to a class rather than a particular
object.
 It is shared between all the objects of this class
 It is defined outside the class constructor.

Class Attributes

# Class attributes
class Shark:
species = 'fish'
def __init__(self, name):
self.name = name
sammy = Shark("Sammy")
print(sammy.species)
stevie = Shark("Stevie")
print(stevie.species)

Instance Methods
 The most common type of methods in Python classes, they can
access unique data of their instance.
 Any method you create will automatically be created as an
instance method.
class MyClass:
def method(self):
return 'instance method called'

Static Methods
 Methods that are related to a class in some way, but don't need to
access any class-specific data.
 Static methods are great for utility functions, which perform a
task in isolation.

Static Methods
 You don't have to use self, and you don't even need to instantiate
an instance
 To define a static method use @staticmethod decorator.
class MyClass:
@staticmethod
def staticmethod():
return 'static method called'

Class Methods
 They have the access to the state of the class, but they can't access
specific instance data.
 It can modify a class state that would apply across all the
instances of the class.
 Class methods don't need self as an argument, but they do need a
parameter called cls.

Class Methods
 Class methods can manipulate the class itself, which is useful
when you're working on complex projects.
 To define a static method use @classmethod decorator.
class MyClass:
@classmethod
def classmethod(cls):
return 'class method called'

Class Method vs Static Method
 A class method takes cls as first parameter while a static method
needs no specific parameters.
 A class method can access or modify class state while a static
method can't access or modify it.
 In general, static methods know nothing about class state. They
are utility type methods that take some parameters and work upon
those parameters. On the other hand class methods must have the
class as a parameter.

Example
 class Person:
driving_age = 16
def __init__(self, name, age):
self.name = name
self.age = age
# Print peron details
def print_details(self):
print(f'name: {self.name}, age: {self.age}')
# A class method to update driving age
@classmethod
def update_driving_age(cls, age):
Person.driving_age = age
# A static method to check if a Person is adult or not.
@staticmethod
def is_adult(age):
return age > 18

Challenge
 Update the previous Person class by adding a method to check if
he/she reaches to the retirement age

Relations between Classes
 Usually classes are related with each others in different ways.
 The basic relations between classes are Inheritance and
Composition.

Inheritance
 One way that object-oriented programming achieves recyclable
code is through inheritance.
 Inheritance is when a class uses features available in another
class.
 Biology, we can think of a child inheriting certain traits from his
parent.

Parent Class and Child Class
 Parent or base classes create a pattern out of which child or subclasses can be
based on.
 Let's say we have a general Bank_account parent class that has
Personal_account and Business_account child.
 Similar methods such as methods to withdraw and deposit money will be
defined in parent class.
 The Business_account subclass would have methods specific to it as well as
variables for example employee_identification_number

Example

class Person:
def __init__(self, person_name, person_age):
self.name = person_name
self.age = person_age
def show_name(self):
print(self.name)
def show_age(self):
print(self.age)

Example

class Student(Person):
def __init__(self, student_name, student_age, student_id):
super().__init__(student_name, student_age)
self.student_id = student_id
def get_id(self):
return self.student_id
person1 = Person("Richard", 23)
person1.show_age()
student1 = Student("Max", 22, "102")
print(student1.get_id())
student1.show_name()

Multiple Inheritance
 Multiple inheritance is when a class can inherit attributes and
methods from more than one parent class.

Example

class TeamMember(object):
def __init__(self, name, uid):
self.name = name
self.uid = uid
class Worker(object):
def __init__(self, pay, jobtitle):
self.pay = pay
self.jobtitle = jobtitle
class TeamLeader(TeamMember, Worker):
def __init__(self, name, uid, pay, jobtitle, exp):
self.exp = exp
TeamMember.__init__(self, name, uid)
Worker.__init__(self, pay, jobtitle)

Example

def print_details(self):
print("name: {}, id: {}, pay: {}, jobtitle: {}, Exp:
{}".format(self.name, self.uid, self.pay, self.jobtitle, self.exp))
tl = TeamLeader('Jake', 10001, 250000, 'Scrum Master', 5)
tl.print_details()

Important Note
 Multiple inheritance allows us to use the code from more than one
parent class in a child class.
 If the same method is defined in multiple parent methods, the
child class will use the method of the first parent declared in its
tuple list.

Challenge
 Create a class employee
Class Attribute: number of employees
Object Attributes: name, salary
Methods: Display number of employees
 Create a class for full time employees inherits the previous class
Instance Attribute: number of days for leave
Methods: Display employee details

Composition
 Composition is a "whole-part" relationship between two objects.
 One of which is composed of one or more instances of the other.

Example

class Professor:
def __init__(self, name, age, address):
self.name = name
self.age = age
self.address = address
def __str__(self):
return f'ph.{self.name}, age: {self.age}, address: {self.address}'
class Department:
def __init__(self, name, prof):
self.name = name
self.prof = prof
def __str__(self):
return f'Department of {self.name}, {self.prof}'

Example

prof = Professor('Edward', 56, 'London')
dep = Department('Science', prof)
print(dep)

Challenges
 We have the following classes
• Person class: has name and address.
• Employ class: has salary, id, and position.
• Department class: has a head of department, department name,
and list of employees, as well as the ability to promote an
employee.
 Define the attributes and methods of each class.
 Define the relationships between these classes.

Summary
 What is OOP?
 Important Definitions
 Attributes Types
 Methods Types
 Relations between Classes

Day 3
 OS Module
 Working with files
 Working with JSON Data
 Web Scraping (BeautifulSoup)

Outlines
 OS Module
 OS Function
 Walking Directory Tree
 OS.path Functions

OS Module
 This module provides a way of using operating system dependent
functionality.
 If you want to manipulate paths, see the os.path module
import os
dir(os)

OS Functions
 os.mkdir(dir_name): create directory.
 os.makedirs(path): create directory and sub directories.
 os.rmdir(dir_name): remove directory
 os.removedirs(path): remove all directories in the path.

OS Functions
 os.getcwd(): get current working directory.
 os.chdir(path): change working directory.
 os.listdir(path): get list of files and directories.
 os.rename(orginal_name, new_name): rename a file.

Walking Directory Tree
 Example:
import os
for dirPath, dirNames, fileNames in
os.walk('C:UsersSamerDesktop'):
print('Current path : ', dirPath)
print('Directories : ', dirNames)
print('Files : ', fileNames)
print('')

Challenge
 Create a directory with your name
 Create other directories inside your folder
 Create some files in different folders
 Print walking tree directory

Environment Variables

os.environ
os.environ.get('PATH')

Launching External Commands
 Example:
import os
cmd = "git --version"
returned_value = os.system(cmd)

OS.path Functions
 os.path.join(path1, path2): joining two paths
 os.path.basename(path): get the file name
 os.path.dirname(path): get the folder name
 os.path.split(path): get the folder and file names

OS.path Functions
 os.path.exists(path): check of existence
 os.path.isdir(path): check if directory
 os.path.splitext(path): split to name and extension

Challenge
 Create a directory with the name 'images'
 Create inside the previous directory two directories with the
names 'train' and 'test'
 Create the following files in the previous two directories
(im1.JPG -> im100.JPG)
 Rename all of the images (im1.jpg -> im100.jpg)

Summary
 OS Module
 OS Function
 Walking Directory Tree
 OS.path Functions

Outlines
 Create a file
 Reading a file
 Updating a file
 Delete a file

Create a File
 Example:
# Opening
fo = open("foo.txt", "w")
# Writing
fo.write( "Python is a great language.nYeah its great!!n")
# Closing
fo.close()

Modes of Opening a File
 r
Opens a file for reading only. The file pointer is placed at the
beginning of the file. This is the default mode.
 w
Opens a file for writing only. Overwrites the file if the file exists. If
the file does not exist, creates a new file for writing.
 a
Opens a file for appending. The file pointer is at the end of the file
if the file exists. That is, the file is in the append mode. If the file
does not exist, it creates a new file for writing.

Read a File
 Example:
# Opening
fo = open("foo.txt", "r")
# Reading
str1 = fo.read()
print ("Read String is : ", str1)
# Closing
fo.close()

File Attributes
 Example:
fo = open("foo.txt", "w")
print ("Name of the file: ", fo.name)
print ("Closed or not : ", fo.closed)
print ("Opening mode : ", fo.mode)
fo.close()

Rename a File
 Syntax:
import os
os.rename(current_file_name, new_file_name)

Deleting File
 Syntax:
import os
os.remove(file_name)

Challenge
 Create a file and write your full name in this file
 Append to this file your best friend name

Challenge
 Create a file and write the following three sentences:
 Write a program to get the file content and then count how many
times each word occurs and how many times each letter occurs.
I live in KL
KL is a great city
You have to visit KL at least one time in your life

Summary
 Create a file
 Reading a file
 Updating a file
 Delete a file

Module 10
Working with JSON Data

Outlines
 What is JSON
 Valid Data Types
 Serialization and Deserialization

JSON
 JavaScript Object Notation was inspired by a subset of the
JavaScript programming language dealing with object literal
syntax
 It is a standardized format the community uses to pass data
around
 JSON has quickly become the de facto standard for information
exchange

JSON Example

JSON = {
"firstName": "Jane",
"lastName": "Doe",
"hobbies": ["running", "sky diving", "singing"],
"age": 35,
"children": [
{
"firstName": "Alice",
"age": 6
},
{
"firstName": "Bob",
"age": 8
}
]
}

JSON Valid Data Types
 In JSON, values must be one of the following data types:
• a string
• a number
• an object (JSON object)
• an array
• a boolean
• null

Serialization and Deserialization
 The process of encoding JSON is usually called Serialization
 Deserialization is the reciprocal process of decoding data that has
been stored or delivered in the JSON standard.

Serializing JSON
 Syntax:
 Check also: dump
from json import *
js_string = dumps(js)

Deserializing JSON
 Syntax:
 Check also: load
from json import *
js = loads(json_string)

Challenge
 Create a JSON file of the name 'data.json' which should
Contain the following data (Serialization)
 Load the data from the file 'data.json' (Deserialization)

Summary
 What is JSON
 Valid Data Types
 Serialization and Deserialization

Module 11
Web Scraping (BeautifulSoup)

Outlines
 BeautifulSoup Installation
 Python Web Scraping
 Web Page Components
 Web Page Scraping Phases
 Searching by Class or Id
 Nested Elements

BeautifulSoup Installation
 Standard Python distribution doesn't come bundled with
BeautifulSoup module.
 In order to install, you have to open the command prompt and
type
 If you use Anaconda, it is already installed
pip install beautifulsoup4

Python Web Scraping
 Web scraping : getting data from a web page into a format you
can work with in your analysis.
 Python gives us BeautifulSoup and other libraries to do web
scraping
 Requirements : before starting with web scraping, we must make
a review of..
The components of any web page

Web Page Components
 HTML — contain the main content of the page.
 CSS — add styling to make the page look nicer.
 JS — Javascript files add interactivity to web pages.
What we have to care about is HTML ?

HTML
 HTML: is the main components of any web page.
 HTML consists of elements called tags.
<html> </html> : the main tag.
<head> </head>
<body> </body>
 : defines a paragraph

HTML Tags
 a : a tags are links
 div : indicates a division, or area, of the page.
 b : bolds any text inside.
 i : italicizes any text inside.
 table : creates a table.
 form : creates an input form.

Relationships Between Tags
 child : a child is a tag inside another tag.
 parent : a parent is the tag another tag is inside.
 sibling : a sibling is a tag that is nested inside the same parent as
another tag

HTML Tags Properties
 Class and id: these special properties give HTML elements
names, and make them easier to interact with when we're
scraping.
 One element can have multiple classes, and a class can be shared
between elements.
 Each element can only have one id, and an id can only be used
once on a page.
 Classes and ids are optional, and not all elements will have them.

Web Page Scraping Phases (Phase 1)
 Phase 1 : Get the web page
 This object has a status_code attribute, which indicates if the page was
downloaded successfully
 To print the web page content
import requests
page=requests.get('http://dataquestio.github.io/web-scraping-pages/simple.html')
page.status_code (200 means successful request)
print(page.text)
print(page.content)

Web Page Scraping Phases (Phase 2)
 Phase2: Parse the page with BeautifulSoup
 To print the web page content
from bs4 import BeautifulSoup
soup = BeautifulSoup(page.content, 'html.parser')
print(soup.prettify())

Example
 Open the following page:
 Let us extract the paragraph text in the page
http://dataquestio.github.io/web-scraping-pages/simple.html

Example (The Long Way)
 Select all the elements at the top level of the page using the
children property of soup, which returns a list generator:
list(soup.children)

 Find the type of each element in the list:
 All of the items are BeautifulSoup objects
[type(item) for item in list(soup.children)]
[bs4.element.Doctype,bs4.element.NavigableString, bs4.element.Tag]

• Select the html tag and its children:
• Find the children inside the html tag
html = list(soup.children)[2]
list(html.children)

 Extract the text inside the p tag :
 Get the p tag by finding the children of the body tag:
 Isolate the p tag:
body = list(html.children)[3]
list(body.children)
p = list(body.children)[1]

 Extract all of the text inside the tag using the get_text method:
 'Here is some simple content for this page.'
p.get_text()

Example (The Short Way)
 Find all the instances of a tag on a page using find_all method
which returns a list:
 Extract text by using list indexing:
soup.find_all('p')
soup.find_all('p')[0].get_text()

Example (The Short Way)
 We have also 'find' method
soup.find('p')

Searching by Class or Id
 Use class and id when scraping to specify specific elements we
want to scrape

Example
 Assume we have: <html>
<head>
<title>A simple example page</title>
</head>
<body>
<div>

First paragraph.


Second paragraph.

</div>

Example



First outer paragraph.




Second outer paragraph.


</body>
</html>

Example
 Phase 1:
 Phase 2:
Page=requests.get("http://dataquestio.github.io/web-scraping-
pages/ids_and_classes.html")

Example
 Use the find_all method to search for items by class or by id:
 Look for any tag that has the class outer-text:
 Search for elements by id:
soup.find_all('p', class_='outer-text')
soup.find_all(class_="outer-text")
soup.find(id="first")

Nested Elements
 We use soup.select(" NESTED ELEMENTS ") to look for
information inside nested elements
finds all a tags inside of a p tag
p a
finds all a tags inside of a p tag inside of abody tag.
body p a
finds all body tags inside of an html tag.
html body
finds all p tags with a class of outer-text.
p.outer-text
finds all p tags with an id of first.
p#first
finds any p tags with a class of outer-text inside of a
body tag.
body p.outer-
text

Example:
 Downloading weather data from the following page:
http://forecast.weather.gov/MapClick.php?lat=37.7772&l
on=-122.4168

Challenge:
 Open the following page:
 Extract the following information from the page:
1- Number of tweets
2- Number of following
3- Number of followers
4- Number of likes
https://twitter.com/realDonaldTrump

Challenge
 Open a text file and write the following lines:
 Extract the same previously mentioned information for all of
these accounts
realdonaldtrump
vp
hillaryclinton

Summary
 BeautifulSoup Installation
 Python Web Scraping
 Web Page Components
 Web Page Scraping Phases
 Searching by Class or Id
 Nested Elements

Day 4
 Introduction to Matrix Processing (NumPy)
 Data Analysis (Pandas Module)
 Data Visualization (Matplotlib)

Module 12
Introduction to Matrix Processing
(NumPy)

Outlines
 What is Numpy?
 NumPy Features
 NumPy Installation
 NumPy - Ndarray Object
 Array Creation
 Elementwise Operations

Outlines
 Broadcasting
 Array Functions
 Array Indexing
 Shape Manipulation
 Copies

What is Numpy?
 NumPy is the fundamental Python package for scientific
computing where it contains:
A powerful N-dimensional array class
Useful linear algebra, Fourier transform, and random number
capabilities
 NumPy is open source.

NumPy Installation
 Standard Python distribution doesn't come bundled with NumPy
module.
type
pip install numpy

NumPy - Ndarray Class
 Note: NumPy's main object is the homogeneous
multidimensional array.
 It is a table of elements (usually numbers), all of the same type
 In NumPy dimensions are called axes.
 For example, the coordinates of a point in 3D space [1,2,3] has
one axis. That axis has 3 elements in it, so we say it has a length
of 3 .

 In the example pictured below, the array has 2 axes. The first axis
has a length of 2 ,the second axis has a length of 3 :
[ [1,2,3] ,
[4,5,6] ]

 NumPy's array class is called ndarray. It is also known by the alias
array.
 Note that numpy.array is not the same as the Standard Python
Library class array.array, which only handles one-dimensional
arrays and offers less functionality.

Ndarray Attribute
 ndarray.ndim: the number of axes (dimensions) of the array.
 ndarray.shape: the dimensions of the array.
 ndarray.size: the total number of elements inside the array

Ndarray Attribute
 ndarray.dtype: the type of the elements in the array.
 ndarray.itemsize: the size in bytes of each element of the array.
 ndarray.data: the actual elements of the array.

Ndarray Attribute (Example)

import numpy as np
data = [[1,2,3] , [4,5,6]]
a = np.array(data)
print('a =' ,a)
print('the shape of my array is = ' , a.shape)
print('the Dimensional of my array is = ' , a.ndim)
print('the size in bytes of each element of my array is = ', a.itemsize)
print('the total number of elements of my array is = ', a.size)
print('the type of the elements in my array is ' , type(a))

Data Type (dtype)
 A data type object describes interpretation of fixed block of
memory corresponding to an array, depending on the following
aspects.
 Size of data (8 – 16 – 32 …. )

Data Type (dtype)
 Type of data:
• bool
• int8, uint8, int16, uint16, int64, uint64
• float16,float32,float64
• complex64, complex128, complex256
• string

Array Creation
 You can convert a one-dimensional or two-dimensional list of
data to an array by calling the array() NumPy function.
# list of data
data1 = [11, 22, 33, 44, 55]
data2 = [[11, 22, 33], [44, 55, 66]]
# array of data
data1 = numpy.array(data1)
data2 = numpy.array(data2)

Array Creation Functions
 np.zeros: Create an array full of zeros.
 np.ones: Create an array full of ones.
 np.empty: Create array with some default values.
 np.arange: Create array of sequences numbers.

Challenge
 Create a numpy array from a list.
• Import the "numpy" library as "np".
• Create a list with values 1 to 10 and assign it to the variable "x".
• Create an integer array from "x" and assign it to the variable "a1".
• Create an array of floats from "x" and assign it to the variable
"a2".
• Print the data type of each array (use the attribute "dtype").

Challenge
 Create arrays in different ways.
• Create an array of shape (2, 3, 4) of zeros.
• Create an array of shape (2, 3, 4) of ones.
• Create an array with values 0 to 999

Elementwise Operations
 Sum, Subtract, division and Elementwise Product are all done as
elementwise operations.
 Also there is logical operation such as and, or.

Broadcasting
 Broadcasting describes how arrays with different shapes are
treated during arithmetic operations.
 Understanding element-wise operations and the same shape
requirement provide a basis for the concept of broadcasting and
why it is used.

Broadcasting Rules
 In order to broadcast, the size of the trailing axes for both arrays in an
operation must either be the same size or one of them must be one.
<=TRUE=>
FALSE

Challenge
 Perform some array calculations.
• Create a 2-D array of shape (2, 4) containing two lists (range(4),
range(10, 14)) and assign it to the variable "a".
• Create an array from a list [2, -1, 1, 0] and assign it to the variable
"b".
• Multiply array "a" by "b" and view the result. Do you understand
how NumPy has used its broadcasting feature to do the
calculation even though the arrays are different shapes?

Challenge
 Perform some array calculations.
• Multiply array "b" by 100 and assign the result to the variable
"b1".
• Multiply array "b" by 100.0 and assign the result to the variable
"b2".

Universal Functions "ufun"
 Sum Function
 Sum By Rows And By Columns
 Other Universal Functions

Computing Sums
 Many array operations, such as computing the sum of all the
elements in the array, are implemented as methods of the ndarray
class.
 Sum an array's Elements :
from numpy import array
var = np.array( [1, 3, 5] )
print('var = ' , var)
s = var.sum()
print('Sum result = ' , s)

Basic Universal Functions
 Extrema:
x = np.array([1, 3, 2])
x.min()
x.max()
x.argmin()
x.argmax()

 Statistics :
x = np.array([1, 2, 3, 1])
y = np.array([[1, 2, 3], [5, 6, 1]])
x.mean()
np.median(x)
np.median(y, axis=-1) # last axis
x.std() # full population standard dev.

 Logical operations:
np.all([True, True, False])
np.any([True, True, False])

 Note : Can be used for array comparisons:
a = np.zeros((100, 100))
np.any(a != 0)
np.all(a == a)
a = np.array([1, 2, 3, 2])
b = np.array([2, 2, 3, 2])
c = np.array([6, 4, 4, 5])
((a <= b) & (b <= c)).all()

Challenge
 Create an array of values 0 to 9 and assign it to the variable "arr".
 Is there any element less than 4?
 Is there any element bigger than 10?

Indexing and Slicing
 Array Indexing
 Indexing With Boolean Arrays
 Array Slicing

Array Indexing
 Once your data is represented using a NumPy array, you can
access it using indexing.
 You can access elements using the bracket operator [ ].
 Specifying integers too large for the bound of the array will cause
an error.
 We can use negative indexes to retrieve values offset from the end
of the array.

Array Indexing
 Example:
data = array([11, 22, 33, 44, 55])
data1 = array([[11, 22], [33, 44]])
print(data[2])
print(data1[0][1])

Array Slicing
 Structures like lists and NumPy arrays can be sliced. This means
that a subsequence of the structure can be indexed and retrieved.
 You can use positive or negative values in slicing.
data[from:to:step]

Shape Manipulation
 Changing the shape of an array
 Stacking together different arrays
 Splitting one array into several smaller one

Changing the Shape of an Array
 ravel(): returns the array as flattened array.
 arr.T: returns the transposed array.
 reshape(): returns an array with modified shape.
Note: the previous three functions return a modified view of the
array, But they do not change the original array.

Stacking Together Different Arrays
 Several arrays can be stacked together along different axes:
• np.hstack : stack arrays in sequence horizontally (column wise).
• np.vstack : stack arrays in sequence vertically (row wise).

Splitting One Array into Several
Smaller Ones
 np.hsplit : split an array into multiple sub-arrays horizontally
(column-wise).
 np.vsplit : split an array into multiple sub-arrays horizontally
(row-wise).

Challenge
 Generate new arrays by modifying our array.
• Create a 2-D array of shape (2, 4) containing two lists (range(4),
range(10, 14)) and assign it to the variable "arr".
• Print the array re-shaped to (2, 2, 2).
• Print the array transposed.
• Print the array flattened to a single dimension.

Copies
 View or Shallow Copy
 Deep Copy

Shallow Copy
 When operating and manipulating arrays, their data is sometimes
copied into a new array and sometimes not.
 Simple assignments make no copy of array objects or of their
data.
 Achieving shallow copy using simple assignment
a = b

Deep Copy
 Deep copy is a process in which the copying process occurs
recursively.
 It means first constructing a new collection object and then
recursively populating it with copies of the child objects found in
the original.
b = a.copy()

Summary
 What is Numpy?
 NumPy Features
 NumPy Installation
 NumPy - Ndarray Object
 Array Creation
 Elementwise Operations

Summary
 Broadcasting
 Array Functions
 Array Indexing
 Shape Manipulation
 Copies

Module 13
Data Analysis (Pandas)

Outlines
 What is Pandas?
 Series
 Dataframe
 Data Importing
 Duplicates
 Missing Values

Outlines
 Dropping
 Filtering
 Sorting
 Grouping
 Resample
 Data concatenation

What is Pandas?
 Pandas provides a comprehensive set of data structures for
working with and manipulating data and performing various
statistical and financial analysis.
 The two primary data structures in Pandas is Series and
DataFrame

What is Pandas?
Columns names
Indexes
Column
Row
Values

Pandas Installation
 Standard Python distribution doesn't come bundled with Pandas
module.
type
pip install pandas

Series
 Series is the primary building block of pandas
 It represents a one-dimensional labeled NumPy array

Creating a Series from a List
 Example:
import pandas as pd
print (pd.Series([1,3,5,6]))
print(pd.Series([1,3,5,6],index=['A1','A2','A3','A4']))

Creating a Series from a NP Array
 Example:
import numpy as np
import pandas as pd
a = np.random.randn(100)*5+100
date =pd.date_range('20170101',periods=100)
s = pd.Series(a,index=date)
print (s)

Creating a Series from a Dictionary
 Example:
import pandas as pd
a = {'A1':5,'A2':3,'A3':6,'A4':2}
s = pd.Series(a)
print (a)

Series Arithmetic Operators
 Series arithmetic matches the index
a = pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd'])
b = pd.Series([4, 3, 2, 1], index=['d', 'c', 'b', 'a'])
a + b # different from Python list
a - b
a * b
a/b

Series Attributes
 s.index: show the indexes
 s.values: show the values
 len(s): number of elements

Viewing Series Data
 s.head(): first 5 rows
 s.head(10): first 10 rows
 s.tail(): last 5 rows
 s.tail(10): last 10 rows

Selecting Data
 s['b'] or s.loc['b']: by named label
 s[2] or s.iloc[2]: by integer index
 s[['b','d']]: multiple select by label
 s[[1,3]]: multiple select by index integer
 s[3:8]: slice items 3 to 8
 Question: What is the difference between s[2] and s[[2]]?

Challenge
date = pd.date_range('20170101',periods=20)
s = pd.Series(np.random.randn(20),index=date)
 Slice out the data from 2017-01-05 to 2017-01-10

DataFrame
 DataFrame is similar to a spreadsheet in Excel
 It has rows index and columns name

Creating a DataFrame from a List
 Example:
import pandas as pd
d = [[1,2],[3,4]]
df= pd.DataFrame(d,index=[1,2],columns=['a','b'])
print (df)

Creating a DataFrame from a NumPy Array
 Example:
import pandas as pd
import numpy as np
d = np.arange(24).reshape(6,4)
df = pd.DataFrame(d,
index=np.arange(1,7),
columns=list('ABCD'))
print (df)

Creating a DataFrame from a Dictionary
 Example:
import pandas as pd
print (pd.DataFrame(
{
'name': ['Ally','Jane','Belinda'],
'height':[160,155,163],
},
columns = ['name','height'],
index = ['A1','A2','A3']))

Create a DataFrame from a Series
 Example:
import pandas as pd
date = pd.date_range('20170101',periods=6)
s1 = pd.Series(np.random.randn(6),index=date)
s2 = pd.Series(np.random.randn(6),index=date)
df = pd.DataFrame({'Asia':s1,'Europe':s2})
print (df)

DataFrame Attributes
 df.shape: dimensionality of a DF
 df.columns: columns of a DF
 df.index: index of a DF
 df.values: values of a DF

Challenges
 Create the DataFrame below
name height age
A1 Ally 160 40
A2 Jane 155 35
A3 Belinda 163 42

DataFrame Arithmetic Operators
 Example:
import pandas as pd
a = [[3,4],[5,6]]
b = [[6,5],[4,3]]
a2 = pd.DataFrame(a,index=[1,2],columns=['d','b'])
b2 = pd.DataFrame(b,index=[3,2],columns=['c','b'])
print(a2+b2)

Viewing Data
 Example:
import pandas as pd
a =
pd.DataFrame(np.random.randn(20,5))
print (a.head())
print (a.head(5))
print (a.tail())
print (a.tail(5))

Select Column Data
 Example:
df[['name','height']]
df.name
df[[1,2]]

Select Row Data
 Example:
print(df.iloc[0])
print(df.loc['A1'])

Slicing Row Data
 Example:
df.iloc[1:3]
df.iloc[1:]
df.iloc[:3]

Selecting Scalar Data
 Example:
df.loc['A1']['height']
df.iloc[1]['height']

Import Excel Data
 Using read_excel function
sp500=pd.read_excel('sp500.xlsx')
sp500.head

Import SQL Data
 Example:
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine('sqlite://', echo=False)
pd.read_csv('train.csv').to_sql('titanic', con=engine)
df_ = pd.read_sql_table('titanic', engine)
df_.head()

Import CSV Data
 Let's read "'Social Influence on Shopping" dataset Using
read_csv:
 This dataset contain a survey of 2,676 millennials: What social
platform has influenced your online shopping the most?
df = pd.read_csv('Social Influence on Shopping.csv')
df.head()

Challenge
 Read the following dataset (twitter_airline_sentiment.csv) and
explore the different dataset attributes.

Handling Duplicates
 To maintain accurate and consistent datasets
 To avoid producing erroneous and misleading statistics

Handling Duplicates
 df.duplicated(): generates a series with number of elements equal
to number of rows in the dataframe where: :
• True: if the row is duplicates
• False: if the row is not duplicates
 df.drop_duplicates(): to drop the rows that have duplicates in the
data frame
 df.drop_duplicates('column name'): returns a dataframe without
duplicates in the specified column

Handling Duplicates
 Example:
df = pd.DataFrame([[2, 2, 2, 0],
[3, 3, 4, 4],
[3, 8, 5, 5],
[3, 3, 4, 4]],
columns = ['A', 'B', 'C', 'D'])
df.duplicated()
df.drop_duplicates()
df.drop_duplicates('A')

Challenge
 How many duplicated tweets we have?
 Remove the duplicated tweets

Missing Values in Python
 By default, missing values are represented with NaN (not a
number)

Missing Values in Python
 Discovering what's missing
 Filling in for the missing values
 Counting missing values
 Filtering out missing values

Discovering what’s Missing
 df.isnull(): give each cell in data frame the following value:
• True: if the cell value is NAN
• False: if the cell value is not NAN

Filling in the Missing Data
 df.fillna(val): replace all NAN with val
 df.fillna({column: val}): replace NAN in column with val
 df.fillna(method='ffill'): replace all NAN with last valid
observation
 df.fillna(method='bfill'): replace all NAN with next valid
observation

Counting Missing Data
 To count number of missing values in each column:
df.isnull().sum()

Filtering Missing Data
 df.dropna(): remove all rows that contains NAN in some
columns
 df.dropna(axis=1): remove all columns that contains NAN
 df.dropna(how='all'): remove all rows that contains NAN in all
columns
 df.dropna(subset =[column]): remove all rows that contains
NAN in specific column

Challenge
 How many missing values we have in the column
(airline_sentiment)?
 What is the best way to handle the missing values in this case?

Dropping
 df.drop([indexes]): dropping rows from the data frame
 df.drop(columns = [cols]): dropping columns from the data
frame

Challenge
 What are the columns that should be dropped? Drop them if any?

Filtering DataFrame
 Using condition to select rows:
sp500 = pd.read_csv('data/sp500.csv',index_col ='Symbol',
usecols=['Symbol','Sector','Price'])
sp500[sp500.Price > 100]
sp500[sp500.Price == sp500.Price.max()]
sp500[(sp500.Price < 10) & (sp500.Price > 0)]

Filtering DataFrame
 You can also use where function to filter dataframe rows:
cond = sp500.Price > 100
df.where(cond)

Challenge
 Filter out the tweets about 'American' airlines

DataFrame Apply Function
 Pandas provides an member function in Dataframe class to apply
a function along the axis of the Dataframe
df.segment_type.apply(lambda x: 'M' if x == 'Mobile' else x )

Challenge
 Convert all the string in the 'airline' column into lower case

Sorting
 Using sort_values function
sorted_df = df.sort_values(by=['count'])
sorted_df.tail()

Challenge
 Challenge: What is the tweet with highest number of likes?
 Find the top 10 retweeted tweets?

Grouping
Index Fruit
1 Apple
2 Apple
3 Orange
4 Apple
5 Orange
Sub-Group
Apple
Orange

Reasons for Grouping Data
 To compare subsets
 To deduce reasons why subgroups differ

Group by
 gr = df.groupby('column name’): group the data frame based on
the column name
 gr.groups: return a dictionary where groups are keys and the
values are the indexes that belonging to that group
 gr.describe: return summary statistics for all numerical columns.
 gr.get_group(group_name): return specific group as a data
frame.

Group by
 Example:
df = pd.read_csv("data/mtcars.csv",usecols=['car_names','mpg','cyl','hp'])
df2 = df.groupby('cyl')
df2.groups
df2.mean()
df2.max()
df2.sum()
df2.size()

Challenge
 What is the average confidence of tweets sentiment (Positive -
Negative - Neutral)?
 How many times does each sentiment category occur?
 What is the total number of likes for the tweets about (American)
airline?

Resample
 Resampling is primarily used for time series data in which the
Object must have a datetime-like index
(DatetimeIndex, PeriodIndex, or TimedeltaIndex)
df.resample(rule).sum()
df.resample(rule).mean()

Challenge
 How many tweets have been posted every day?
 What is the average confidence of tweets sentiment (Positive -
Negative - Neutral) per week?

Concatenate
 Combine two DataFrame objects with identical columns.
 Columns outside the intersection will be filled with NaN values.
join
axis
ingore_index
keys
pd.concat([df1,df2])

Concatenate with Join
 Concat only have inner and outer joins
pd.concat([df1, df2], join="inner")
pd.concat([df1, df2], join="outer")

Concatenate with Axis
 Combine horizontally along the x axis by passing in axis=1.
 Note :
axis = 0 is index
axis = 1 is columns
The default is axis = 0
df3 = pd.concat([df1, df2], axis=1)

Concatenate with Keys
 Add a hierarchical index at the outermost level of the data
pd.concat([df1, df2], keys=['s1', 's2',])

Concatenate with Ignore Index
 Re-order the index
pd.concat([df1, df2], ignore_index=True)

Insert a New Column
 Example:
df.insert(1,'d',[11,12])

Append 2 DataFrames
 Similar to Concat, append 2nd data frames by rows to the first
data frame
df3 = df1.append(df2, ignore_index=True)

Challenge
 read the 'extra_tweets.csv' and concatenate with the final
dataframe.

Summary
 What is Pandas?
 Series
 Dataframe
 Data Importing
 Duplicates
 Missing Values

Summary
 Dropping
 Filtering
 Sorting
 Grouping
 Resample
 Data concatenation

Module 14
Data Visualization (Matplotlib)

Outlines
 What is Matplotlib?
 Types of Plots
 Graph Details
 Working with Multiple Figures and Axes
 Working with Text
 Logarithmic and other Nonlinear Axes

What is Matplotlib?
 matplotlib.pyplot is a plotting library used for 2D graphics in
python programming language.

Matplotlib Installation
 Standard Python distribution doesn't come bundled with
Matplotlib module.
type
pip install matplotlib

Types of Plots
 There are various graphs which can be created using python
matplotlib.

Line Graph
 Example:
from matplotlib import pyplot as plt
plt.plot([1, 2, 3], [4, 5, 1])
plt.show()

Controlling Line Properties
 Lines have many attributes that you can set: linewidth, dash style,
etc.
plt.plot(x, y, linewidth=2.0)

Multiple Line Graphs
 Example:
import numpy as np
import matplotlib.pyplot as plt
t = np.arange(0., 5., 0.2)
plt.plot(t, t, 'r--', t, t**2, 'bs', t, t**3, 'g^')
plt.show()

Graph Details
 Example:
from matplotlib import pyplot as plt
x = [5, 8, 10]
y = [12, 16, 6]
x2 = [6, 9, 11]
y2 = [6, 15, 7]
plt.plot(x, y, 'g', label='line one', linewidth=5)
plt.plot(x2, y2, 'c', label='line two', linewidth=5)
plt.title('Epic Info')
plt.ylabel('Y axis')
plt.xlabel('X axis')
plt.legend()
plt.grid(True, color='k')
plt.show()

Bar Graph
 Bar graph uses bars to compare data among different categories.
 It is well suited when you want to measure the changes over a
period of time.
 It can be represented horizontally or vertically.
 Also, the important thing to keep in mind is that longer the bar,
greater is the value.

Example

import numpy as np
n_groups = 4
means_frank = (90, 55, 40, 65)
means_guido = (85, 62, 54, 20)
index = np.arange(n_groups)
bar_width = 0.35
opacity = 0.8

Example

rects1 = plt.bar(index, means_frank, bar_width,
alpha=opacity, color='b', abel='Frank')
rects2 = plt.bar(index + bar_width, means_guido,
bar_width, alpha=opacity, color='g', label='Guido')
plt.xlabel('Person')
plt.ylabel('Scores')
plt.title('Scores by person')
plt.xticks(index+bar_width/2, ('A', 'B', 'C', 'D'))
plt.legend()
plt.show()

Histogram
 Histograms are used to show a distribution whereas a bar chart is
used to compare different entities.
 Histograms are useful when you have arrays or a very long list.

Example

population_age=
[22,55,62,45,21,22,34,42,42,4,2,102,95,85,55,110,120,70,
65,55,111,115,80,75,65,54,44,43,42,48]
bins = [0,10,20,30,40,50,60,70,80,90,100]
plt.hist(population_age, bins, rwidth=0.8)
plt.xlabel('age groups')
plt.ylabel('Number of people')
plt.title('Histogram')
plt.show()

Scatter Plot
 We need scatter plots in order to compare variables.
 For example, how much one variable is affected by another
variable to build a relation out of it.
 The data is displayed as a collection of points, each having the
value of one variable which determines the position on the
horizontal axis and the value of other variable determines the
position on the vertical axis.

Example

x = [1, 1.5, 2, 2.5, 3, 3.5, 3.6]
y = [7.5, 8, 8.5, 9, 9.5, 10, 10.5]
x1 = [8, 8.5, 9, 9.5, 10, 10.5, 11]
y1 = [3, 3.5, 3.7, 4, 4.5, 5, 5.2]
plt.scatter(x, y, label='high income low saving', color='r')
plt.scatter(x1, y1, label='low income high savings', color='b')
plt.xlabel('saving')
plt.ylabel('income')
plt.title('Scatter Plot')
plt.legend()
plt.show()

Area Graph
 Area plots are pretty much similar to the line plot.
 They are also known as stack plots.
 These plots can be used to track changes over time for two or
more related groups that make up one whole category.

Example

fig, ax = plt.subplots()
days = [1,2,3,4,5]
sleeping = [7,8,6,11,7]
eating = [2,3,4,3,2]
working = [7,8,7,2,2]
playing = [8,5,7,8,13]
ax.plot([],[],color='m', label='Sleeping', linewidth=5)
ax.plot([],[],color='c', label='Eating', linewidth=5)
plt.show()

Example

ax.plot([],[],color='r', label='Working', linewidth=5)
ax.plot([],[],color='k', label='Playing', linewidth=5)
ax.stackplot(days, sleeping,eating,working,playing,
colors=['m','c','r','k'])
plt.xlabel('x')
plt.ylabel('y')
plt.title('Interesting GraphnCheck it out')
plt.legend()

Pie Chart
 A pie chart refers to a circular graph which is broken down into
segments i.e. slices of pie.
 It is basically used to show the percentage or proportional data
where each slice of pie represents a category.

Example

slices = [7, 2, 2, 13]
activities = ['sleeping', 'eating', 'working', 'playing']
cols = ['c', 'm', 'r', 'b']
plt.figure(figsize=(5,5))
plt.pie(slices,labels=activities,colors=cols,shadow=True,
explode=(0.0, 0.0, 0.0,0.0),
autopct='%1.2f%%')
plt.title('Pie Plot')
plt.show()

Working with Multiple Figures and Axes

import numpy as np
def f(t):
return np.exp(-t) * np.cos(2*np.pi*t)
t1 = np.arange(0.0, 5.0, 0.1)
t2 = np.arange(0.0, 5.0, 0.02)
plt.figure(1)
plt.subplot(211)
plt.plot(t1, f(t1), 'bo', t2, f(t2), 'k')
plt.subplot(212)
plt.plot(t2, np.cos(2*np.pi*t2), 'r--')
plt.show()

Example
 import numpy as np
def f(t):
return np.exp(-t) * np.cos(2 * np.pi * t)
t1 = np.arange(0.0, 5.0, 0.1)
t2 = np.arange(0.0, 5.0, 0.02)
plt.subplot(221)
plt.plot(t1, f(t1), 'bo', t2, f(t2))
plt.subplot(222)
plt.plot(t2, np.cos(2 * np.pi * t2))
plt.subplot(223)
plt.plot(t2, np.cos(2 * np.pi * t2))
plt.subplot(224)
plt.plot(t1, f(t1), 'bo', t2, f(t2))
plt.show()

Working with Text
 import numpy as np
np.random.seed(19680801)
mu, sigma = 100, 15
x = mu + sigma * np.random.randn(10000)
# the histogram of the data
n, bins, patches = plt.hist(x, 50, normed=1, facecolor='g', alpha=0.75)
plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.title('Histogram of IQ')
plt.text(60, .025, 'iTrain')
plt.axis([40, 160, 0, 0.03])
plt.grid(True)
plt.show()

Annotating Text
 The uses of the basic text() command above place text at an
arbitrary position on the Axes.
 A common use for text is to annotate some feature of the plot,
and the annotate() method provides helper functionality to make
annotations easy.
 In an annotation, there are two points to consider: the location
being annotated represented by the argument x, y and the location
of the text x, y text. Both of these arguments are (x,y) tuples.

Example

import numpy as np
t = np.arange(0.0, 5.0, 0.01)
s = np.cos(2*np.pi*t)
plt.plot(t, s, lw=2)
plt.annotate('local max', xy=(2, 1),
xytext=(3, 1.5),
arrowprops=dict(facecolor='black'))
plt.ylim(-2,2)
plt.show()

Logarithmic and other Nonlinear Axes
 Matplotlib.pyplot supports not only linear axis scales, but also
logarithmic scales.
 This is commonly used if data spans many orders of magnitude.
 Changing the scale of an axis is easy:
 plt.xscale('log')

Challenges
• Draw exp function
• Draw log function
• Draw sin and cos together with different colors
• If you have 75 engineers, 50 managers, and 10 accountants, draw
a pie chart to show the percentage of employees

Summary
 What is Matplotlib?
 Types of Plots
 Graph Details
 Working with Multiple Figures and Axes
 Working with Text
 Logarithmic and other Nonlinear Axes

Day 5
 Introduction to Applied Machine Learning
 Capstone Project
 Course Exam

Module 15
Introduction to Applied
Machine Learning

Outlines
• Introduction
• Machine Learning Algorithms Types
• Main Steps in Machine Learning Projects
• Introduction to Scikit Learn Module

Introduction
 Before we go in depth in this Module, here are some important
terms to know about:
Artificial Intelligence , Machine Learning , Deep Learning.

Introduction
 Artificial Intelligence:
The effort to automate intellectual tasks normally performed
by humans, it encompasses machine learning and deep learning.

Introduction
 Machine Learning:

Introduction
 Deep Learning:
Is a specific subfield of machine learning: a new take on learning
representations from data that puts an emphasis on learning
successive layers of increasingly meaningful representations.

How Old?
https://how-old.net/

What's Dog?
https://www.what-dog.net/

Types of Machine Learning
 Supervised Learning
Classification
Regression
 Un-Supervised Learning
Clustering
Dimensional Reduction
 Reinforcement Learning
Q-Learning
Policy Gradient
DQN

Machine Learning Steps
 Get Data
 Clean and Preprocess Data
 Shuffle (Randomize) Data
 Split into Training/Test Data
 Set Model Parameters - Learning Rate, Loss Function, Optimizer
 Training the Model
 Evaluate the Model
 Use the Model

Step 3: Shuffle/Randomize Data

Step 4: Split Training/Test Data
 Training Data
 Test Data

What is Scikits Learn
 Scikit-learn is an open source Python library of popular machine
learning algorithms
 It started in 2007 as a Google Summer of Code project
 It is built upon NumPy and SciPy

SciKit Learn
http://scikit-learn.org/

Scikits Learn Modules
 Classifications
 Regression
 Clustering
 Dimensional Reduction
 Model Selection
 Preprocessing

Summary
 What is Machine Learning
 Machine Learning Algorithms Types
 Main Steps in Machine Learning Projects
 Introduction to Scikit Learn module

Summary
Day 1
• Python Basics
Day 2
• Jupyter Notebook
• Functions in Python
• Pythonic Programming
• Modules and Packages
• Object Oriented Programming

Summary
Day 3
• OS Module
• Working with files
• Working with JSON Data
• Web Scraping (BeautifulSoup)
Day 4:
• Introduction to Matrix Processing (NumPy)
• Data Analysis (Pandas Module)
• Data Visualization (Matplotlib)

Summary
Day 5
• Introduction to Applied Machine Learning
• Capstone Project
• Course Exam

CPPDS Slide.pdf

More Related Content

Similar to CPPDS Slide.pdf (20)

More from Fadlie Ahdon (6)

Recently uploaded (20)

CPPDS Slide.pdf