Fr

ee

NumPy Beginner's Guide Third Edition

NumPy

Third Edition

In today's world of science and technology, it's all about speed and flexibility. When it comes to scientific
computing, NumPy tops the list. NumPy will give you both speed and high productivity. This book will walk
you through NumPy with clear, step-by-step examples and just the right amount of theory. The book focuses
on the fundamentals of NumPy, including array objects, functions, and matrices, each of them explained
with practical examples. You will then learn about different NumPy modules while performing mathematical
operations such as calculating the Fourier transform, finding the inverse of a matrix, and determining eigenvalues,
among many others. This book is a one-stop solution to knowing the ins and outs of the vast NumPy library,
empowering you to use its wide range of mathematical features to build efficient, high-speed programs.

Sa

pl
e

What this book will do

for you...

Use NumPy array objects to perform

array operations
Familiarize yourself with commonly used
NumPy functions

The Beginner's Guide approach...

Use NumPy matrices for matrix algebra

Clear step-by-step instructions for the most

Work with the NumPy modules to perform

various algebraic operations

Learn by doing start working right away

Test NumPy code with the numpy.testing

module

Inspiring, realistic examples that give you

ideas for your own work

community experience distilled

P U B L I S H I N G

Third Edition

Tasks and challenges to encourage

experimentation

$ 44.99 US
29.99 UK

Learn by doing: less theory, more results

NumPy

Leave out the boring bits

Ivan Idris

Plot simple plots, subplots, histograms,

and more with matplotlib

useful tasks

Beginner's Guide

Install NumPy, matplotlib, SciPy, and

IPython on various operating systems

Build efficient, high-speed programs using the

high-performance NumPy mathematical library

Beginner's Guide

Prices do not include

local sales tax or VAT
where applicable

P U B L I S H I N G

Visit www.PacktPub.com for books, eBooks,

code, downloads, and PacktLib.

Ivan Idris

In this package, you will find:

The author biography

A preview chapter from the book, Chapter 3 'Getting Familiar with Commonly
Used Functions'
A synopsis of the books content
More information on NumPy Beginner's Guide Third Edition

About the Author

Ivan Idris has an MSc in experimental physics. His graduation thesis had a strong emphasis
on applied computer science. After graduating, he worked for several companies as a Java
developer, data warehouse developer, and QA Analyst. His main professional interests are
business intelligence, big data, and cloud computing. Ivan enjoys writing clean, testable
code and interesting technical articles. He is the author of NumPy Beginner's Guide, NumPy
Cookbook, Learning NumPy Array, and Python Data Analysis. You can find more information
about him and a blog with a few examples of NumPy at http://ivanidris.net/
wordpress/.

Preface
Scientists, engineers, and quantitative data analysts face many challenges nowadays. Data
scientists want to be able to perform numerical analysis on large datasets with minimal
programming eort. They also want to write readable, ecient, and fast code that is as close
as possible to the mathematical language they are used to. A number of accepted solutions
are available in the scientific computing world.
The C, C++, and Fortran programming languages have their benefits, but they are not
interactive and considered too complex by many. The common commercial alternatives,
such as MATLAB, Maple, and Mathematica, provide powerful scripting languages that are
even more limited than any general-purpose programming language. Other open source
tools similar to MATLAB exist, such as R, GNU Octave, and Scilab. Obviously, they too lack
the power of a language such as Python.
Python is a popular general-purpose programming language that is widely used in the
scientific community. You can access legacy C, Fortran, or R code easily from Python. It
is object-oriented and considered to be of a higher level than C or Fortran. It allows you
to write readable and clean code with minimal fuss. However, it lacks an out-of-the-box
MATLAB equivalent. That's where NumPy comes in. This book is about NumPy and related
Python libraries, such as SciPy and matplotlib.

What is NumPy?
NumPy (short for numerical Python) is an open source Python library for scientific
computing. It lets you work with arrays and matrices in a natural way. The library contains
a long list of useful mathematical functions, including some functions for linear algebra,
Fourier transformation, and random number generation routines. LAPACK, a linear algebra
library, is used by the NumPy linear algebra module if you have it installed on your system.
Otherwise, NumPy provides its own implementation. LAPACK is a well-known library,
originally written in Fortran, on which MATLAB relies as well. In a way, NumPy replaces some
of the functionality of MATLAB and Mathematica, allowing rapid interactive prototyping.

Preface

We will not be discussing NumPy from a developing contributor's perspective, but from more
of a user's perspective. NumPy is a very active project and has a lot of contributors. Maybe,
one day you will be one of them!

History
NumPy is based on its predecessor Numeric. Numeric was first released in 1995 and has
deprecated status now. Neither Numeric nor NumPy made it into the standard Python library
for various reasons. However, you can install NumPy separately, which will be explained in
Chapter 1, NumPy Quick Start.
In 2001, a number of people inspired by Numeric created SciPy, an open source scientific
computing Python library that provides functionality similar to that of MATLAB, Maple, and
Mathematica. Around this time, people were growing increasingly unhappy with Numeric.
Numarray was created as an alternative to Numeric. That is also deprecated now. It was
better in some areas than Numeric, but worked very dierently. For that reason, SciPy kept
on depending on the Numeric philosophy and the Numeric array object. As is customary
with new latest and greatest software, the arrival of Numarray led to the development of
an entire ecosystem around it, with a range of useful tools.
In 2005, Travis Oliphant, an early contributor to SciPy, decided to do something about this
situation. He tried to integrate some of Numarray's features into Numeric. A complete
rewrite took place, and it culminated in the release of NumPy 1.0 in 2006. At that time,
NumPy had all the features of Numeric and Numarray, and more. Tools were available to
facilitate the upgrade from Numeric and Numarray. The upgrade is recommended since
Numeric and Numarray are not actively supported any more.
Originally, the NumPy code was a part of SciPy. It was later separated and is now used by
SciPy for array and matrix processing.

Why use NumPy?

NumPy code is much cleaner than straight Python code and it tries to accomplish the
same tasks. There are fewer loops required because operations work directly on arrays
and matrices. The many convenience and mathematical functions make life easier as well.
The underlying algorithms have stood the test of time and have been designed with high
performance in mind.
NumPy's arrays are stored more eciently than an equivalent data structure in base Python,
such as a list of lists. Array IO is significantly faster too. The improvement in performance
scales with the number of elements of the array. For large arrays, it really pays o to use
NumPy. Files as large as several terabytes can be memory-mapped to arrays, leading to
optimal reading and writing of data.

Preface

The drawback of NumPy arrays is that they are more specialized than plain lists. Outside the
context of numerical computations, NumPy arrays are less useful. The technical details of
NumPy arrays will be discussed in later chapters.
Large portions of NumPy are written in C. This makes NumPy faster than pure Python code.
A NumPy C API exists as well, and it allows further extension of functionality with the help
of the C language. The C API falls outside the scope of the book. Finally, since NumPy is open
source, you get all the related advantages. The price is the lowest possibleas free as a
beer. You don't have to worry about licenses every time somebody joins your team or you
need an upgrade of the software. The source code is available for everyone. This of course is
beneficial to code quality.

Limitations of NumPy
If you are a Java programmer, you might be interested in Jython, the Java implementation of
Python. In that case, I have bad news for you. Unfortunately, Jython runs on the Java Virtual
Machine and cannot access NumPy because NumPy's modules are mostly written in C. You
could say that Jython and Python are two totally dierent worlds, though they do implement
the same specifications. There are some workarounds for this discussed in NumPy Cookbook
- Second Edition, Packt Publishing, written by Ivan Idris.

What this book covers

Chapter 1, NumPy Quick Start, guides you through the steps needed to install NumPy on
your system and create a basic NumPy application.
Chapter 2, Beginning with NumPy Fundamentals, introduces NumPy arrays and
fundamentals.
Chapter 3, Getting Familiar with Commonly Used Functions, teaches you the most commonly
used NumPy functionsthe basic mathematical and statistical functions.
Chapter 4, Convenience Functions for Your Convenience, tells you about functions that
make working with NumPy easier. This includes functions that select certain parts of your
arrays, for instance, based on a Boolean condition. You also learn about polynomials and
manipulating the shapes of NumPy objects.
Chapter 5, Working with Matrices and ufuncs, covers matrices and universal functions.
Matrices are well-known in mathematics and have their representation in NumPy as well.
Universal functions (ufuncs) work on arrays element by element, or on scalars. ufuncs expect
a set of scalars as the input and produce a set of scalars as the output.

Preface

Chapter 6, Moving Further with NumPy Modules, discusses a number of basic modules
of universal functions. These functions can typically be mapped to their mathematical
counterparts, such as addition, subtraction, division, and multiplication.
Chapter 7, Peeking into Special Routines, describes some of the more specialized NumPy
functions. As NumPy users, we sometimes find ourselves having special requirements.
Fortunately, NumPy satisfies most of our needs.
Chapter 8, Assuring Quality with Testing, teaches you how to write NumPy unit tests.
Chapter 9, Plotting with matplotlib, covers matplotlib in depth, a very useful Python plotting
library. NumPy cannot be used on its own to create graphs and plots. matplotlib integrates
nicely with NumPy and has plotting capabilities comparable to MATLAB.
Chapter 10, When NumPy Is Not Enough SciPy and Beyond, covers more details about
SciPy. We know that SciPy and NumPy are historically related. SciPy, as mentioned in the
History section, is a high-level Python scientific computing framework built on top of NumPy.
It can be used in conjunction with NumPy.
Chapter 11, Playing with Pygame, is the dessert of this book. You learn how to create fun
games with NumPy and Pygame. You also get a taste of artificial intelligence in this chapter.
Appendix A, Pop Quiz Answers, has the answers to all the pop quiz questions within
the chapters.
Appendix B, Additional Online Resources, contains links to Python, mathematics, and
statistics websites.
Appendix C, NumPy Functions' References, lists some useful NumPy functions and
their descriptions.

Getting Familiar with Commonly

Used Functions
In this chapter, we will have a look at common NumPy functions. In particular,
we will learn how to load data from files by using an example involving
historical stock prices. Also, we will get to see the basic NumPy mathematical
and statistical functions.
We will learn how to read from and write to files. Also, we will get a taste of the
functional programming and linear algebra possibilities in NumPy.

In this chapter, we shall cover the following topics:

Functions working on arrays

Loading arrays from files

Writing arrays to files

Simple mathematical and statistical functions

File I/O
First, we will learn about file I/O with NumPy. Data is usually stored in files. You would not
get far if you were not able to read from and write to files.

[ 53 ]

Getting Familiar with Commonly Used Functions

Time for action reading and writing files

As an example of file I/O, we will create an identity matrix and store its contents in a file.
In this and other chapters, we will use the following line by convention
to import NumPy:
import numpy as np

Perform the following steps to do so:

1.

The identity matrix is a square matrix with ones on the main diagonal and zeros for
the rest (see https://www.khanacademy.org/math/precalculus/precalcmatrices/zero-identity-matrix-tutorial/v/identity-matrix).
The identity matrix can be created with the eye() function. The only argument that
we need to give the eye() function is the number of ones. So, for instance, for a
two-by-two matrix, write the following code:
i2 = np.eye(2)
print(i2)

The output is:

[[ 1. 0.]
[ 0. 1.]]

2.

Save the data in a plain text file with the savetxt() function. Specify the name of
the file that we want to save the data in and the array containing the data itself:
np.savetxt("eye.txt", i2)

A file called eye.txt should have been created in the same directory as the Python script.

What just happened?

Reading and writing files is a necessary skill for data analysis. We wrote to a file with
savetxt(). We made an identity matrix with the eye() function.
Instead of a filename, we can also provide a file handle. A file handle is a term
in many programming languages, which means a variable pointing to a file, like
a postal address. For more information on how to get a file handle in Python,
please refer to http://www.diveintopython3.net/files.html.

[ 54 ]

Chapter 3

You can check for yourself whether the contents are as expected. The code for this example
can be downloaded from the book support website: https://www.packtpub.com/
books/content/support (see save.py)
import numpy as np
i2 = np.eye(2)
print(i2)
np.savetxt("eye.txt", i2))

Comma-seperated value files

Files in the Comma-seperated value (CSV) format are encountered quite frequently. Often,
the CSV file is just a dump from a database. Usually, each field in the CSV file corresponds to
a database table column. As we all know, spreadsheet programs, such as Excel, can produce
CSV files, as well.

Time for action loading from CSV files

How do we deal with CSV files? Luckily, the loadtxt() function can conveniently read CSV
files, split up the fields, and load the data into NumPy arrays. In the following example, we
will load historical stock price data for Apple (the company, not the fruit). The data is in CSV
format and is part of the code bundle for this book. The first column contains a symbol that
identifies the stock. In our case, it is AAPL. Second is the date in dd-mm-yyyy format. The
third column is empty. Then, in order, we have the open, high, low, and close price. Last,
but not least, is the trading volume of the day. This is what a line looks like:
AAPL,28-01-2011, ,344.17,344.4,333.53,336.1,21144800

For now, we are only interested in the close price and volume. In the preceding sample, that
will be 336.1 and 21144800. Store the close price and volume in two arrays as follows:
c,v=np.loadtxt('data.csv', delimiter=',', usecols=(6,7), unpack=True)

As you can see, data is stored in the data.csv file. We have set the delimiter to, (comma),
since we are dealing with a CSV file. The usecols parameter is set through a tuple to get
the seventh and eighth fields, which correspond to the close price and volume. The unpack
argument is set to True, which means that data will be unpacked and assigned to the c and
v variables that will hold the close price and volume, respectively.

[ 55 ]

Getting Familiar with Commonly Used Functions

Volume Weighted Average Price

Volume Weighted Average Price (VWAP) is a very important quantity in finance. It represents
an average price for a financial asset (see https://www.khanacademy.org/math/
probability/descriptive-statistics/old-stats-videos/v/statistics-theaverage). The higher the volume, the more significant a price move typically is. VWAP is

often used in algorithmic trading and is calculated using volume values as weights.

Time for action calculating Volume Weighted Average Price

The following are the actions that we will take:

1.
2.

Read the data into arrays.

Calculate VWAP:
from __future__ import print_function
import numpy as np
c,v=np.loadtxt('data.csv', delimiter=',', usecols=(6,7),
unpack=True)
vwap = np.average(c, weights=v)
print("VWAP =", vwap)

The output is as follows:

VWAP = 350.589549353

What just happened?

That wasn't very hard, was it? We just called the average() function and set its weights
parameter to use the v array for weights. By the way, NumPy also has a function to calculate
the arithmetic mean. This is an unweighted average with all the weights equal to 1.

The mean() function

The mean() function is quite friendly and not so mean. This function calculates the
arithmetic mean of an array.

[ 56 ]

Chapter 3

The arithmetic mean is given by the following formula:

1 n
ai
n i =1
It sums the values in an array a and divides the sum by the number
of elements n (see https://www.khanacademy.org/math/
probability/descriptive-statistics/central_
tendency/e/mean_median_and_mode).

Let's see it in action:

print("mean =", np.mean(c))

As a result, we get the following printout:

mean = 351.037666667

Time-weighted average price

In finance, time-weighted average price (TWAP) is another average price measure. Now that
we are at it, let's compute the TWAP too. It is just a variation on a theme really. The idea is
that recent price quotes are more important, so we should give recent prices higher weights.
The easiest way is to create an array with the arange() function of increasing values from
zero to the number of elements in the close price array. This is not necessarily the correct
way. In fact, most of the examples concerning stock price analysis in this book are only
illustrative. The following is the TWAP code:
t = np.arange(len(c))
print("twap =", np.average(c, weights=t))

It produces the following output:

twap = 352.428321839

The TWAP is even higher than the mean.

Pop quiz computing the weighted average

Q1. Which function returns the weighted average of an array?
1.
2.
3.
4.

weighted average
waverage
average
avg
[ 57 ]

Getting Familiar with Commonly Used Functions

Have a go hero calculating other averages

Try doing the same calculation using the open price. Calculate the mean for the volume and
the other prices.

Value range
Usually, we don't only want to know the average or arithmetic mean of a set of values,
which are in the middle, to know we also want the extremes, the full rangethe highest and
lowest values. The sample data that we are using here already has those values per daythe
high and low price. However, we need to know the highest value of the high price and the
lowest price value of the low price.

Time for action finding highest and lowest values

The min() and max() functions are the answer for our requirement. Perform the following
steps to find the highest and lowest values:

1.

First, read our file again and store the values for the high and low prices into arrays:
h,l=np.loadtxt('data.csv', delimiter=',', usecols=(4,5),
unpack=True)

The only thing that changed is the usecols parameter, since the high and low
prices are situated in dierent columns.

2.

The following code gets the price range:

print("highest =", np.max(h))
print("lowest =", np.min(l))

These are the values returned:

highest = 364.9
lowest = 333.53

Now, it's easy to get a midpoint, so it is left as an exercise for you to attempt.

3.

NumPy allows us to compute the spread of an array with a function called ptp().
The ptp() function returns the dierence between the maximum and minimum
values of an array. In other words, it is equal to max(array)min(array). Call
the ptp() function:
print("Spread high price", np.ptp(h))
print("Spread low price", np.ptp(l))

[ 58 ]

Chapter 3

You will see this text printed:

Spread high price 24.86
Spread low price 26.97

What just happened?

We defined a range of highest to lowest values for the price. The highest value was given by
applying the max() function to the high price array. Similarly, the lowest value was found
by calling the min() function to the low price array. We also calculated the peak-to-peak
distance with the ptp() function:
from __future__ import print_function
import numpy as np
h,l=np.loadtxt('data.csv', delimiter=',', usecols=(4,5), unpack=True)
print("highest =", np.max(h))
print("lowest =", np.min(l))
print((np.max(h) + np.min(l)) /2)
print("Spread high price", np.ptp(h))
print("Spread low price", np.ptp(l))

Statistics
Stock traders are interested in the most probable close price. Common sense says that
this should be close to some kind of an average as the price dances around a mean, due to
random fluctuations. The arithmetic mean and weighted average are ways to find the center
of a distribution of values. However, neither are robust and both are sensitive to outliers.
Outliers are extreme values that are much bigger or smaller than the typical values in a
dataset. Usually, outliers are caused by a rare phenomenon or a measurement error. For
instance, if we have a close price value of a million dollars, this will influence the outcome
of our calculations.

Time for action performing simple statistics

We can use some kind of threshold to weed out outliers, but there is a better way. It is called
the median, and it basically picks the middle value of a sorted set of values (see https://
www.khanacademy.org/math/probability/descriptive-statistics/central_
tendency/e/mean_median_and_mode). One half of the data is below the median and the

other half is above it. For example, if we have the values of 1, 2, 3, 4, and 5, then the median
will be 3, since it is in the middle.

[ 59 ]

Getting Familiar with Commonly Used Functions

These are the steps to calculate the median:

1.

Create a new Python script and call it simplestats.py. You already know how to
load the data from a CSV file into an array. So, copy that line of code and make sure
that it only gets the close price. The code should appear like this:
c=np.loadtxt('data.csv', delimiter=',', usecols=(6,), unpack=True)

2.

The function that will do the magic for us is called median(). We will call it and
print the result immediately. Add the following line of code:
print("median =", np.median(c))

The program prints the following output:

median = 352.055

3.

Since it is our first time using the median() function, we would like to check
whether this is correct. Obviously, we can do it by just going through the file and
finding the correct value, but that is no fun. Instead, we will just mimic the median
algorithm by sorting the close price array and printing the middle value of the sorted
array. The msort() function does the first part for us. Call the function, store the
sorted array, and then print it:
sorted_close = np.msort(c)
print("sorted =", sorted_close)

This prints the following output:

Yup, it works! Let's now get the middle value of the sorted array:
N = len(c)
print "middle =", sorted[(N - 1)/2]

The preceding snippet gives us the following output:

middle = 351.99

4.

Hey, that's a dierent value than the one the median() function gave us. How
come? Upon further investigation, we find that the median() function return value
doesn't even appear in our file. That's even stranger! Before filing bugs with the
NumPy team, let's have a look at the documentation:
$ python
>>> import numpy as np
>>> help(np.median)
[ 60 ]

Chapter 3

This mystery is easy to solve. It turns out that our naive algorithm only works for
arrays with odd lengths. For even-length arrays, the median is calculated from the
average of the two array values in the middle. Therefore, type the following code:
print("average middle =", (sorted[N /2] + sorted[(N - 1) / 2]) /
2)

This prints the following output:

average middle = 352.055

5.

Another statistical measure that we are concerned with is variance. Variance tells
us how much a variable varies (see https://www.khanacademy.org/math/
probability/descriptive-statistics/variance_std_deviation/e/
variance). In our case, it also tells us how risky an investment is, since a stock

price that varies too wildly is bound to get us into trouble.

Calculate the variance of the close price (with NumPy, this is just a one-liner):
print("variance =", np.var(c))

This gives us the following output:

variance = 50.1265178889

6.

Not that we don't trust NumPy or anything, but let's double-check using the
definition of variance, as found in the documentation. Mind you, this definition
might be dierent than the one in your statistics book, but that is quite common
in the field of statistics.
The population variance is defined as the mean
of the square of deviations from the mean, divided by the
number of elements in the array:

1 n
2
a

mean
(
)

i
n i =1
Some books tell us to divide by the number of elements in the array minus one (this
is called a sample variance):
print("variance from definition =", np.mean((c - c.mean())**2))

The output is as follows:

variance from definition = 50.1265178889

[ 61 ]

Getting Familiar with Commonly Used Functions

What just happened?

Maybe you noticed something new. We suddenly called the mean() function on the c
array. Yes, this is legal, because the ndarray class has a mean() method. This is for your
convenience. For now, just keep in mind that this is possible. The code for this example
can be found in simplestats.py:
from __future__ import print_function
import numpy as np
c=np.loadtxt('data.csv', delimiter=',', usecols=(6,), unpack=True)
print("median =", np.median(c))
sorted = np.msort(c)
print("sorted =", sorted)
N = len(c)
print("middle =", sorted[(N - 1)/2])
print("average middle =", (sorted[N /2] + sorted[(N - 1) / 2]) / 2)
print("variance =", np.var(c))
print("variance from definition =", np.mean((c - c.mean())**2))

Stock returns
In academic literature, it is more common to base analysis on stock returns and log returns
of the close price. Simple returns are just the rate of change from one value to the next.
Logarithmic returns, or log returns, are determined by taking the log of all the prices and
calculating the dierences between them. In high school, we learned that:

a
log ( a ) log ( b ) = log
b
Log returns, therefore, also measure the rate of change. Returns are dimensionless, since,
in the act of dividing, we divide dollar by dollar (or some other currency). Anyway, investors
are most likely to be interested in the variance or standard deviation of the returns, as this
represents risk.

[ 62 ]

Chapter 3

Time for action analyzing stock returns

Perform the following steps to analyze stock returns:

1.

First, let's calculate simple returns. NumPy has the diff() function that returns an
array that is built up of the dierence between two consecutive array elements. This
is sort of like dierentiation in calculus (the derivative of price with respect to time).
To get the returns, we also have to divide by the value of the previous day. We must
be careful though. The array returned by diff() is one element shorter than the
close prices array. After careful deliberation, we get the following code:
returns = np.diff( arr ) / arr[ : -1]

Notice that we don't use the last value in the divisor. The standard deviation is
equal to the square root of variance. Compute the standard deviation using the
std() function:
print("Standard deviation =", np.std(returns))

This results in the following output:

Standard deviation = 0.0129221344368

2.

The log return or logarithmic return is even easier to calculate. Use the log()
function to get the natural logarithm of the close price and then unleash the
diff() function on the result:
logreturns = np.diff(np.log(c))

Normally, we have to check that the input array doesn't have zeros or negative
numbers. If it does, we will get an error. Stock prices are, however, always positive,
so we didn't have to check.

3.

Quite likely, we will be interested in days when the return is positive. In the current
setup, we can get the next best thing with the where() function, which returns the
indices of an array that satisfies a condition. Just type the following code:
posretindices = np.where(returns > 0)
print("Indices with positive returns", posretindices)

This gives us a number of indices for the array elements that are positive as a tuple,
recognizable by the round brackets on both sides of the printout:
Indices with positive returns (array([ 0, 1, 4, 5,
10, 11, 12, 16, 17, 18, 19, 21, 22, 23, 25, 28]),)

[ 63 ]

6,

7,

9,

Getting Familiar with Commonly Used Functions

4.

In investing, volatility measures price variation of a financial security. Historical

volatility is calculated from historical price data. The logarithmic returns are
interesting if you want to know the historical volatilityfor instance, the annualized
or monthly volatility. The annualized volatility is equal to the standard deviation of
the log returns as a ratio of its mean, divided by one over the square root of the
number of business days in a year, usually one assumes 252. Calculate it with the
std() and mean() functions, as in the following code:
annual_volatility = np.std(logreturns)/np.mean(logreturns)
annual_volatility = annual_volatility / np.sqrt(1./252.)
print(annual_volatility)

Take notice of the division within the sqrt() function. Since, in Python, integer
division works dierently than float division, we needed to use floats to make
sure that we get the proper results. The monthly volatility is similarly given by
the following code:
print("Monthly volatility", annual_volatility * np.sqrt(1./12.))

What just happened?

We calculated the simple stock returns with the diff() function, which calculates
dierences between sequential elements. The log() function computes the natural
logarithms of array elements. We used it to calculate the logarithmic returns. At the
end of this section, we calculated the annual and monthly volatility (see returns.py):
from __future__ import print_function
import numpy as np
c=np.loadtxt('data.csv', delimiter=',', usecols=(6,), unpack=True)
returns = np.diff( c ) / c[ : -1]
print("Standard deviation =", np.std(returns))
logreturns = np.diff( np.log(c) )
posretindices = np.where(returns > 0)
print("Indices with positive returns", posretindices)
annual_volatility = np.std(logreturns)/np.mean(logreturns)
annual_volatility = annual_volatility / np.sqrt(1./252.)
print("Annual volatility", annual_volatility)
print("Monthly volatility", annual_volatility * np.sqrt(1./12.))

[ 64 ]

Chapter 3

Dates
Do you sometimes have the Monday blues or Friday fever? Ever wondered whether
the stock market suers from these phenomena? Well, I think this certainly warrants
extensive research.

Time for action dealing with dates

First, we will read the close price data. Second, we will split the prices according to the day
of the week. Third, for each weekday, we will calculate the average price. Finally, we will
find out which day of the week has the highest average and which has the lowest average.
A word of warning before we commence: you might be tempted to use the result to buy
stock on one day and sell on the other. However, we don't have enough data to make this
kind of decisions.
Coders hate dates because they are so complicated! NumPy is very much oriented toward
floating point operations. For this reason, we need to take extra eort to process dates. Try it
out yourself; put the following code in a script or use the one that comes with this book:
dates, close=np.loadtxt('data.csv', delimiter=',',
usecols=(1,6), unpack=True)

Execute the script and the following error will appear:

ValueError: invalid literal for float(): 28-01-2011

Now, perform the following steps to deal with dates:

1.

Obviously, NumPy tried to convert the dates into floats. What we have to do is tell
NumPy explicitly how to convert the dates. The loadtxt() function has a special
parameter for this purpose. The parameter is called converters and is a dictionary
that links columns with the so-called converter functions. It is our responsibility to
write the converter function. Write the function down:
# Monday 0
# Tuesday 1
# Wednesday 2
# Thursday 3
# Friday 4
# Saturday 5
# Sunday 6
def datestr2num(s):
return datetime.datetime.strptime(s, "%d-%m-%Y").date().
weekday()

[ 65 ]

Getting Familiar with Commonly Used Functions

We give the datestr2num() function dates as a string, such as 28-01-2011. The

string is first turned into a datetime object, using a specified format %d-%m-%Y. By
the way, this is standard Python and is not related to NumPy itself (see https://
docs.python.org/2/library/datetime.html#strftime-and-strptimebehavior). Second, the datetime object is turned into a day. Finally, the weekday

method is called on the date to return a number. As you can read in the comments,
the number is between 0 and 6. 0 is, for instance, Monday, and 6 is Sunday. The actual
number, of course, is not important for our algorithm; it is only used as identification.

2.

Now, hook up our date converter function:

dates, close=np.loadtxt('data.csv', delimiter=',', usecols=(1,6),
converters={1: datestr2num}, unpack=True)
print "Dates =", dates

This prints the following output:

Dates = [ 4. 0. 1. 2. 3. 4. 0. 1. 2. 3. 4. 0. 1.
3. 4. 1. 2. 4. 0. 1. 2. 3. 4. 0. 1. 2. 3. 4.]

2.

No Saturdays and Sundays, as you can see. Exchanges are closed over the weekend.

3.

We will now make an array that has five elements for each day of the week. Initialize
the values of the array to 0:
averages = np.zeros(5)

This array will hold the averages for each weekday.

4.

We already learned about the where() function that returns indices of the array
for elements that conform to a specified condition. The take() function can use
these indices and takes the values of the corresponding array items. We will use the
take() function to get the close prices for each weekday. In the following loop,
we go through the date values 0 to 4, better known as Monday to Friday. We get
the indices with the where() function for each day and store it in the indices
array. Then, we retrieve the values corresponding to the indices, using the take()
function. Finally, compute an average for each weekday and store it in the averages
array, like this:
for i in range(5):
indices = np.where(dates == i)
prices = np.take(close, indices)
avg = np.mean(prices)
print("Day", i, "prices", prices, "Average", avg)
averages[i] = avg

[ 66 ]

Chapter 3

The loop prints the following output:

Day 0 prices [[ 339.32
351.79

351.88

359.18

353.21

355.36]] Average

Day 1 prices [[ 345.03

Average 350.635

355.2

359.9

338.61

349.31

355.76]]

Day 2 prices [[ 344.32

Average 352.136666667

358.16

363.13

342.62

352.12

352.47]]

Day 3 prices [[ 343.44

Average 350.898333333

354.54

358.3

342.88

359.56

346.67]]

356.85

350.56

348.16

360.

Day 4 prices [[ 336.1

346.5
351.99]] Average 350.022857143

5.

If you want, you can go ahead and find out which day has the highest average, and
which the lowest. However, it is just as easy to find this out with the max() and
min() functions, as shown here:
top = np.max(averages)
print("Highest average", top)
print("Top day of the week", np.argmax(averages))
bottom = np.min(averages)
print("Lowest average", bottom)
print("Bottom day of the week", np.argmin(averages))

The output is as follows:

Highest average 352.136666667
Top day of the week 2
Lowest average 350.022857143
Bottom day of the week 4

What just happened?

The argmin() function returned the index of the lowest value in the averages array.
The index returned was 4, which corresponds to Friday. The argmax() function returned
the index of the highest value in the averages array. The index returned was 2, which
corresponds to Wednesday (see weekdays.py):
from __future__ import print_function
import numpy as np
from datetime import datetime
# Monday 0
# Tuesday 1
# Wednesday 2
[ 67 ]

Getting Familiar with Commonly Used Functions

# Thursday 3
# Friday 4
# Saturday 5
# Sunday 6
def datestr2num(s):
return datetime.strptime(s, "%d-%m-%Y").date().weekday()
dates, close=np.loadtxt('data.csv', delimiter=',', usecols=(1,6),
converters={1: datestr2num}, unpack=True)
print("Dates =", dates)
averages = np.zeros(5)
for i in range(5):
indices = np.where(dates == i)
prices = np.take(close, indices)
avg = np.mean(prices)
print("Day", i, "prices", prices, "Average", avg)
averages[i] = avg

top = np.max(averages)
print("Highest average", top)
print("Top day of the week", np.argmax(averages))
bottom = np.min(averages)
print("Lowest average", bottom)
print("Bottom day of the week", np.argmin(averages))

Have a go hero looking at VWAP and TWAP

Hey, that was fun! For the sample data, it appears that Friday is the cheapest day and
Wednesday is the day when your Apple stock will be worth the most. Ignoring the fact that
we have very little data, is there a better method to compute the averages? Shouldn't we
involve volume data as well? Maybe it makes more sense to you to do a time-weighted
average. Give it a go! Calculate the VWAP and TWAP. You can find some hints on how to go
about doing this at the beginning of this chapter.

[ 68 ]

Chapter 3

Time for action using the datetime64 data type

The datetime64 data type was introduced in NumPy 1.7.0 (see http://docs.scipy.
org/doc/numpy/reference/arrays.datetime.html).

1.

To learn about the datetime64 data type, start a Python shell and import NumPy
as follows:
$ python
>>> import numpy as np

Create a datetime64 from a string (you can use another date if you like):
>>> np.datetime64('2015-04-22')
numpy.datetime64('2015-04-22')

In the preceding code, we created a datetime64 for April 22, 2015, which happens
to be Earth Day. We used the YYYY-MM-DD format, where Y corresponds to the year,
M corresponds to the month, and D corresponds to the day of the month. NumPy
uses the ISO 8601 standard (see http://en.wikipedia.org/wiki/ISO_8601).
This is an international standard to represent dates and times. ISO 8601 allows the
YYYY-MM-DD, YYYY-MM, and YYYYMMDD formats. Check for yourself, as follows:
>>> np.datetime64('2015-04-22')
numpy.datetime64('2015-04-22')
>>> np.datetime64('2015-04')
numpy.datetime64('2015-04')

2.

By default, ISO 8601 uses the local time zone. Times can be specified using the
format T[hh:mm:ss]. For example, define January 1, 1677 at 8:19 p.m. as follows:
>>> local = np.datetime64('1677-01-01T20:19')
>>> local
numpy.datetime64('1677-01-01T20:19Z')

Additionally, a string in the format [hh:mm] specifies an oset that is relative to the
UTC time zone. Create a datetime64 with 9 hours oset, as follows:
>>> with_offset = np.datetime64('1677-01-01T20:19-0900')
>>> with_offset
numpy.datetime64('1677-01-02T05:19Z')

The Z at the end stands for Zulu time, which is how UTC is sometimes referred to.
Subtract the two datetime64 objects from each other:
>>> local - with_offset
numpy.timedelta64(-540,'m')
[ 69 ]

Getting Familiar with Commonly Used Functions

The subtraction creates a NumPy timedelta64 object, which in this case, indicates
a 540 minute dierence. We can also add or subtract a number of days to a
datetime64 object. For instance, April 22, 2015 happens to be a Wednesday. With
the arange() function, create an array holding all the Wednesdays from April 22,
2015 until May 22, 2015 as follows:
>>> np.arange('2015-04-22', '2015-05-22', 7, dtype='datetime64')
array(['2015-04-22', '2015-04-29', '2015-05-06', '2015-05-13',
'2015-05-20'], dtype='datetime64[D]')

Note that in this case, it is mandatory to specify the dtype argument, otherwise
NumPy thinks that we are dealing with strings.

What just happened?

We learned about the NumPy datetime64 type. This data type allows us to manipulate
dates and times with ease. Its features include simple arithmetic and creation of arrays
using the normal NumPy capabilities.

Weekly summary
The data that we used in the previous Time for action section is end-of-day data. In essence,
it is summarized data compiled from the trade data for a certain day. If you are interested in
the market and have decades of data, you might want to summarize and compress the data
even further. Let's summarize the data of Apple stocks to give us weekly summaries.

Time for action summarizing data

The data we will summarize will be for a whole business week, running from Monday
to Friday. During the period covered by the data, there was one holiday on February 21,
President's Day. This happened to be a Monday and the US stock exchanges were closed on
this day. As a consequence, there is no entry for this day, in the sample. The first day in the
sample is a Friday, which is inconvenient. Use the following instructions to summarize data:

1.

To simplify, just have a look at the first three weeks in the sample later, you can
have a go at improving this:
close = close[:16]
dates = dates[:16]

We will be building on the code from the previous Time for action section.

[ 70 ]

Chapter 3

2.

Commencing, we will find the first Monday in our sample data. Recall that Mondays
have the code 0 in Python. This is what we will put in the condition of the where()
function. Then, we will need to extract the first element that has index 0. The result
will be a multidimensional array. Flatten this with the ravel() function:
# get first Monday
first_monday = np.ravel(np.where(dates == 0))[0]
print("The first Monday index is", first_monday)

This will print the following output:

The first Monday index is 1

3.

The next logical step is to find the Friday before last Friday in the sample. The
logic is similar to the one for finding the first Monday, and the code for Friday is 4.
Additionally, we are looking for the second to last element with index 2:
# get last Friday
last_friday = np.ravel(np.where(dates == 4))[-2]
print("The last Friday index is", last_friday)

This will give us the following output:

The last Friday index is 15

4.

Next, create an array with the indices of all the days in the three weeks:
weeks_indices = np.arange(first_monday, last_friday + 1)
print("Weeks indices initial", weeks_indices)

5.

Split the array in pieces of size 5 with the split() function:

weeks_indices = np.split(weeks_indices, 3)
print("Weeks indices after split", weeks_indices)

This splits the array as follows:

Weeks indices after split [array([1, 2, 3, 4, 5]), array([ 6,
8, 9, 10]), array([11, 12, 13, 14, 15])]

6.

7,

In NumPy, array dimensions are called axes. Now, we will get fancy with the apply_
along_axis() function. This function calls another function, which we will provide,
to operate on each of the elements of an array. Currently, we have an array with three
elements. Each array item corresponds to one week in our sample and contains indices
of the corresponding items. Call the apply_along_axis() function by supplying
the name of our function, called summarize(), which we will define shortly.
Furthermore, specify the axis or dimension number (such as 1), the array to operate
on, and a variable number of arguments for the summarize() function, if any:
weeksummary = np.apply_along_axis(summarize, 1,
weeks_indices, open, high, low, close)
print("Week summary", weeksummary)
[ 71 ]

Getting Familiar with Commonly Used Functions

7.

For each week, the summarize() function returns a tuple that holds the open,
high, low, and close price for the week, similar to end-of-day data:
def summarize(a, o, h, l, c):
monday_open = o[a[0]]
week_high = np.max( np.take(h, a) )
week_low = np.min( np.take(l, a) )
friday_close = c[a[-1]]
return("APPL", monday_open, week_high,
week_low, friday_close)

Notice that we used the take() function to get the actual values from indices.
Calculating the high and low values for the week was easily done with the max()
and min() functions. The open for the week is the open for the first day in the
weekMonday. Likewise, the close is the close for the last day of the weekFriday:
Week summary [['APPL' '335.8' '346.7' '334.3' '346.5']
['APPL' '347.89' '360.0' '347.64' '356.85']
['APPL' '356.79' '364.9' '349.52' '350.56']]

8.

Store the data in a file with the NumPy savetxt() function:

np.savetxt("weeksummary.csv", weeksummary, delimiter=",",
fmt="%s")

As you can see, have specified a filename, the array we want to store, a delimiter
(in this case a comma), and the format we want to store floating point numbers in.
The format string starts with a percent sign. Second is an optional flag. Theflag
means left justify, 0 means left pad with zeros, and + means precede with + or -.
Third is an optional width. The width indicates the minimum number of characters.
Fourth, a dot is followed by a number linked to precision. Finally, there comes a
character specifier; in our example, the character specifier is a string. The character
codes are described as follows:
Character code
c

Description

d or i

signed decimal integer

e or E

scientific notation with e or E.

decimal floating point

g,G

use the shorter of e,E or f

signed octal

character

[ 72 ]

Chapter 3

Character code
s

Description

unsigned decimal integer

x,X

unsigned hexadecimal integer

string of characters

View the generated file in your favorite editor or type at the command line:
$ cat weeksummary.csv
APPL,335.8,346.7,334.3,346.5
APPL,347.89,360.0,347.64,356.85
APPL,356.79,364.9,349.52,350.56

What just happened?

We did something that is not even possible in some programming languages. We defined a
function and passed it as an argument to the apply_along_axis() function.
The programming paradigm described here is called functional programming.
You can read more about functional programming in Python at
https://docs.python.org/2/howto/functional.html.

Arguments for the summarize() function were neatly passed by apply_along_axis()

(see weeksummary.py):
from __future__ import print_function
import numpy as np
from datetime import datetime
# Monday 0
# Tuesday 1
# Wednesday 2
# Thursday 3
# Friday 4
# Saturday 5
# Sunday 6
def datestr2num(s):
return datetime.strptime(s, "%d-%m-%Y").date().weekday()
dates, open, high, low, close=np.loadtxt('data.csv', delimiter=',',
usecols=(1, 3, 4, 5, 6), converters={1: datestr2num}, unpack=True)
close = close[:16]
dates = dates[:16]

[ 73 ]

Getting Familiar with Commonly Used Functions

# get first Monday
first_monday = np.ravel(np.where(dates == 0))[0]
print("The first Monday index is", first_monday)
# get last Friday
last_friday = np.ravel(np.where(dates == 4))[-1]
print("The last Friday index is", last_friday)
weeks_indices = np.arange(first_monday, last_friday + 1)
print("Weeks indices initial", weeks_indices)
weeks_indices = np.split(weeks_indices, 3)
print("Weeks indices after split", weeks_indices)
def summarize(a, o, h, l, c):
monday_open = o[a[0]]
week_high = np.max( np.take(h, a) )
week_low = np.min( np.take(l, a) )
friday_close = c[a[-1]]
return("APPL", monday_open, week_high, week_low, friday_close)
weeksummary = np.apply_along_axis(summarize, 1, weeks_indices, open,
high, low, close)
print("Week summary", weeksummary)
np.savetxt("weeksummary.csv", weeksummary, delimiter=",", fmt="%s")

Have a go hero improving the code

Change the code to deal with a holiday. Time the code to see how big the speedup due to
apply_along_axis() is.

Average True Range

The Average True Range (ATR) is a technical indicator that measures volatility of stock prices.
The ATR calculation is not important further but will serve as an example of several NumPy
functions, including the maximum() function.

[ 74 ]

Chapter 3

Time for action calculating the Average True Range

To calculate the ATR, perform the following steps:

1.

The ATR is based on the low and high price of N days, usually the last 20 days.
N = 5
h = h[-N:]
l = l[-N:]

2.

We also need to know the close price of the previous day:

previousclose = c[-N -1: -1]

For each day, we calculate the following:

The daily rangethe dierence between the high and low price:
h l

The dierence between the high and previous close:

h previousclose

The dierence between the previous close and the low price:
previousclose l

3.

The max() function returns the maximum of an array. Based on those three values,
we calculate the so-called true range, which is the maximum of these values. We are
now interested in the element-wise maxima across arraysmeaning the maxima of
the first elements in the arrays, the second elements in the arrays, and so on. Use
the NumPy maximum() function instead of the max() function for this purpose:
truerange = np.maximum(h - l, h - previousclose, previousclose l)

4.

Create an atr array of size N and initialize its values to 0:

atr = np.zeros(N)

5.

The first value of the array is just the average of the truerange array:
atr[0] = np.mean(truerange)

Calculate the other values with the following formula:

( ( N 1) PATR + TR )
N

[ 75 ]

Getting Familiar with Commonly Used Functions

Here, PATR is the previous day's ATR; TR is the true range:

for i in range(1, N):
atr[i] = (N - 1) * atr[i - 1] + truerange[i]
atr[i] /= N

What just happened?

We formed three arrays, one for each of the three rangesdaily range, the gap between the
high of today and the close of yesterday, and the gap between the close of yesterday and the
low of today. This tells us how much the stock price moved and, therefore, how volatile it is.
The algorithm requires us to find the maximum value for each day. The max() function that
we used before can give us the maximum value within an array, but that is not what we want
here. We need the maximum value across arrays, so we want the maximum value of the first
elements in the three arrays, the second elements, and so on. In preceding Time for action
section, we saw that the maximum() function can do this. After this, we computed a moving
average of the true range values (see atr.py):
from __future__ import print_function
import numpy as np
h, l, c = np.loadtxt('data.csv', delimiter=',', usecols=(4, 5, 6),
unpack=True)
N = 5
h = h[-N:]
l = l[-N:]
print("len(h)", len(h), "len(l)", len(l))
print("Close", c)
previousclose = c[-N -1: -1]
print("len(previousclose)", len(previousclose))
print("Previous close", previousclose)
truerange = np.maximum(h - l, h - previousclose, previousclose - l)
print("True range", truerange)
atr = np.zeros(N)
atr[0] = np.mean(truerange)

[ 76 ]

Chapter 3
for i in range(1, N):
atr[i] = (N - 1) * atr[i - 1] + truerange[i]
atr[i] /= N
print("ATR", atr)

In the following sections, we will learn better ways to calculate moving averages.

Have a go hero taking the minimum() function for a spin

Besides the maximum() function, there is a minimum() function. You can probably guess
what it does. Make a small script or start an interactive session in IPython to test your
assumptions.

Simple Moving Average

The Simple Moving Average (SMA) is commonly used to analyze time-series data. To
calculate it, we define a moving window of N periods, N days in our case. We move this
window along the data and calculate the mean of the values inside the window.

Time for action computing the Simple Moving Average

The moving average is easy enough to compute with a few loops and the mean() function,
but NumPy has a better alternativethe convolve() function. The SMA is, after all,
nothing more than a convolution with equal weights or, if you like, unweighted.
Convolution is a mathematical operation on two functions defined as the
integral of the product of the two functions after one of the functions is
reversed and shifted.

( f g )( t ) = f ( ) g ( t ) d = f ( t ) g ( ) d
Convolution is described on Wikipedia at https://en.wikipedia.org/
wiki/Convolution. Khan Academy also has a tutorial on convolution
at https://www.khanacademy.org/math/differentialequations/laplace-transform/convolution-integral/v/
introduction-to-the-convolution.

[ 77 ]

Getting Familiar with Commonly Used Functions

Use the following steps to compute the SMA:

1.

Use the ones() function to create an array of size N and elements initialized to 1,
and then, divide the array by N to give us the weights:
N = 5
weights = np.ones(N) / N
print("Weights", weights)

For N = 5, this gives us the following output:

Weights [ 0.2

2.

0.2

0.2

0.2

0.2]

Now, call the convolve() function with these weights:

c = np.loadtxt('data.csv', delimiter=',', usecols=(6,),
unpack=True)
sma = np.convolve(weights, c)[N-1:-N+1]

3.

From the array returned by convolve(), we extracted the data in the center of size
N. The following code makes an array of time values and plots with matplotlib
that we will cover in a later chapter:
c = np.loadtxt('data.csv', delimiter=',', usecols=(6,),
unpack=True)
sma = np.convolve(weights, c)[N-1:-N+1]
t = np.arange(N - 1, len(c))
plt.plot(t, c[N-1:], lw=1.0, label="Data")
plt.plot(t, sma, '--', lw=2.0, label="Moving average")
plt.title("5 Day Moving Average")
plt.xlabel("Days")
plt.ylabel("Price ($)")
plt.grid()
plt.legend()
plt.show()

[ 78 ]

Chapter 3

In the following chart, the smooth dashed line is the 5 day SMA and the jagged thin
line is the close price:

What just happened?

We computed the SMA for the close stock price. It turns out that the SMA is just a signal
processing techniquea convolution with weights 1/N, where N is the size of the moving
average window. We learned that the ones() function can create an array with ones and
the convolve() function calculates the convolution of a dataset with specified weights
(see sma.py):
from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
N = 5
weights = np.ones(N) / N
print("Weights", weights)
c = np.loadtxt('data.csv', delimiter=',', usecols=(6,), unpack=True)
sma = np.convolve(weights, c)[N-1:-N+1]
t = np.arange(N - 1, len(c))
plt.plot(t, c[N-1:], lw=1.0, label="Data")
plt.plot(t, sma, '--', lw=2.0, label="Moving average")
[ 79 ]

Getting Familiar with Commonly Used Functions

plt.title("5 Day Moving Average")
plt.xlabel("Days")
plt.ylabel("Price ($)")
plt.grid()
plt.legend()
plt.show()

Exponential Moving Average

The Exponential Moving Average (EMA) is a popular alternative to the SMA. This method
uses exponentially decreasing weights. The weights for points in the past decrease
exponentially but never reach zero. We will learn about the exp() and linspace()
functions while calculating the weights.

Time for action calculating the Exponential Moving Average

Given an array, the exp() function calculates the exponential of each array element. For
example, look at the following code:
x = np.arange(5)
print("Exp", np.exp(x))

It gives the following output:

Exp [

1.

2.71828183

7.3890561

20.08553692

54.59815003]

The linspace() function takes as parameters a start value, a stop value, and optionally an
array size. It returns an array of evenly spaced numbers. This is an example:
print("Linspace", np.linspace(-1, 0, 5))

This will give us the following output:

Linspace [-1.

-0.75 -0.5

-0.25

0.

Calculate the EMA for our data:

1.

Now, back to the weights, calculate them with exp() and linspace():
N = 5
weights = np.exp(np.linspace(-1., 0., N))

2.

Normalize the weights with the ndarray sum() method:

weights /= weights.sum()
print("Weights", weights)

[ 80 ]

Chapter 3

For N = 5, we get these weights:

Weights [ 0.11405072
0.31002201]

3.

0.14644403

0.18803785

0.24144538

After this, use the convolve() function that we learned about in the SMA section
and also plot the results:
c = np.loadtxt('data.csv', delimiter=',', usecols=(6,),
unpack=True)
ema = np.convolve(weights, c)[N-1:-N+1]
t = np.arange(N - 1, len(c))
plt.plot(t, c[N-1:], lw=1.0, label='Data')
plt.plot(t, ema, '--', lw=2.0, label='Exponential Moving Average')
plt.title('5 Days Exponential Moving Average')
plt.xlabel('Days')
plt.ylabel('Price ($)')
plt.legend()
plt.grid()
plt.show()

This gives us a nice chart where, again, the close price is the thin jagged line and the
EMA is the smooth dashed line:

[ 81 ]

Getting Familiar with Commonly Used Functions

What just happened?

We calculated the EMA of the close price. First, we computed exponentially decreasing
weights with the exp() and linspace() functions. The linspace() function gave us
an array with evenly spaced elements, and, then, we calculated the exponential for these
numbers. We called the ndarray sum() method in order to normalize the weights. After
this, we applied the convolve() trick that we learned in the SMA section (see ema.py):
from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
x = np.arange(5)
print("Exp", np.exp(x))
print("Linspace", np.linspace(-1, 0, 5))
# Calculate weights
N = 5
weights = np.exp(np.linspace(-1., 0., N))
# Normalize weights
weights /= weights.sum()
print("Weights", weights)
c = np.loadtxt('data.csv', delimiter=',', usecols=(6,), unpack=True)
ema = np.convolve(weights, c)[N-1:-N+1]
t = np.arange(N - 1, len(c))
plt.plot(t, c[N-1:], lw=1.0, label='Data')
plt.plot(t, ema, '--', lw=2.0, label='Exponential Moving Average')
plt.title('5 Days Exponential Moving Average')
plt.xlabel('Days')
plt.ylabel('Price ($)')
plt.legend()
plt.grid()
plt.show()

Bollinger Bands
Bollinger Bands are yet another technical indicator. Yes, there are thousands of them. This
one is named after its inventor and indicates a range for the price of a financial security. It
consists of three parts:
1. A Simple Moving Average.

[ 82 ]

Chapter 3

2. An upper band of two standard deviations above this moving averagethe

standard deviation is derived from the same data with which the moving average
is calculated.
3. A lower band of two standard deviations below the moving average.

Time for action enveloping with Bollinger Bands

We already know how to calculate the SMA. So, if you need to refresh your memory, please
review the Time for action computing the simple average section in this chapter. This
example will introduce the NumPy fill() function. The fill() function sets the value of
an array to a scalar value. The function should be faster than array.flat = scalar or
setting the values of the array one-by-one in a loop. Perform the following steps to envelope
with the Bollinger Bands:

1.

Starting with an array called sma that contains the moving average values, we
will loop through all the datasets corresponding to those values. After forming
the dataset, calculate the standard deviation. Note that at a certain point, it
will be necessary to calculate the dierence between each data point and the
corresponding average value. If we do not have NumPy, we will loop through these
points and subtract each of the values one-by-one from the corresponding average.
However, the NumPy fill() function allows us to construct an array that has
elements set to the same value. This enables us to save on one loop and subtract
arrays in one go:
deviation = []
C = len(c)
for i in range(N - 1, C):
if i + N < C:
dev = c[i: i + N]
else:
dev = c[-N:]
averages = np.zeros(N)
averages.fill(sma[i - N - 1])
dev = dev - averages
dev = dev ** 2
dev = np.sqrt(np.mean(dev))
deviation.append(dev)
deviation = 2 * np.array(deviation)
print(len(deviation), len(sma))
upperBB = sma + deviation
lowerBB = sma - deviation
[ 83 ]

Getting Familiar with Commonly Used Functions

2.

To plot, we will use the following code (don't worry about it now; we will see how
this works in Chapter 9, Plotting with matplotlib):
t = np.arange(N - 1, C)
plt.plot(t, c_slice, lw=1.0, label='Data')
plt.plot(t, sma, '--', lw=2.0, label='Moving Average')
plt.plot(t, upperBB, '-.', lw=3.0, label='Upper Band')
plt.plot(t, lowerBB, ':', lw=4.0, label='Lower Band')
plt.title('Bollinger Bands')
plt.xlabel('Days')
plt.ylabel('Price ($)')
plt.grid()
plt.legend()
plt.show()

Following is a chart showing the Bollinger Bands for our data. The jagged thin line in
the middle represents the close price, and the dashed, smoother line crossing it is
the moving average:

[ 84 ]

Chapter 3

What just happened?

We worked out the Bollinger Bands that envelope the close price of our data. More
importantly, we got acquainted with the NumPy fill() function. This function fills
an array with a scalar value. This is the only parameter of the fill() function (see
bollingerbands.py):
from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
N = 5
weights = np.ones(N) / N
print("Weights", weights)
c = np.loadtxt('data.csv', delimiter=',', usecols=(6,), unpack=True)
sma = np.convolve(weights, c)[N-1:-N+1]
deviation = []
C = len(c)
for i in range(N - 1, C):
if i + N < C:
dev = c[i: i + N]
else:
dev = c[-N:]
averages = np.zeros(N)
averages.fill(sma[i - N - 1])
dev = dev - averages
dev = dev ** 2
dev = np.sqrt(np.mean(dev))
deviation.append(dev)
deviation = 2 * np.array(deviation)
print(len(deviation), len(sma))
upperBB = sma + deviation
lowerBB = sma - deviation
c_slice = c[N-1:]
between_bands = np.where((c_slice < upperBB) & (c_slice > lowerBB))
print(lowerBB[between_bands])
print(c[between_bands])
print(upperBB[between_bands])
[ 85 ]

Getting Familiar with Commonly Used Functions

between_bands = len(np.ravel(between_bands))
print("Ratio between bands", float(between_bands)/len(c_slice))
t = np.arange(N - 1, C)
plt.plot(t, c_slice, lw=1.0, label='Data')
plt.plot(t, sma, '--', lw=2.0, label='Moving Average')
plt.plot(t, upperBB, '-.', lw=3.0, label='Upper Band')
plt.plot(t, lowerBB, ':', lw=4.0, label='Lower Band')
plt.title('Bollinger Bands')
plt.xlabel('Days')
plt.ylabel('Price ($)')
plt.grid()
plt.legend()
plt.show()

Have a go hero switching to Exponential Moving Average

It is customary to choose the SMA to center the Bollinger Band on. The second most popular
choice is the EMA, so try that as an exercise. You can find a suitable example in this chapter,
if you need pointers.
Check whether the fill() function is faster or is as fast as array.flat = scalar, or
setting the value in a loop.

Linear model
Many phenomena in science have a related linear relationship model. The NumPy linalg
package deals with linear algebra computations. We will begin with the assumption that a
price value can be derived from N previous prices based on a linear relationship relation.

Time for action predicting price with a linear model

Keeping an open mind, let's assume that we can express a stock price p as a linear
combination of previous values, that is, a sum of those values multiplied by certain
coecients we need to determine:

pt = b + i =1 at i pt i
N

[ 86 ]

Chapter 3

In linear algebra terms, this boils down to finding a least-squares method (see https://
www.khanacademy.org/math/linear-algebra/alternate_bases/orthogonal_
projections/v/linear-algebra-least-squares-approximation).
Independently of each other, the astronomers Legendre and
Gauss created the least squares method around 1805 (see
http://en.wikipedia.org/wiki/Least_squares).
The method was initially used to analyze the motion of celestial
bodies. The algorithm minimizes the sum of the squared residuals
(the dierence between measured and predicted values):
n

( measured
i =1

predictedi )

The recipe goes as follows:

1.

First, form a vector b containing N price values:

b = c[-N:]
b = b[::-1]
print("b", x)

The result is as follows:

b [ 351.99

2.

346.67

352.47

355.76

355.36]

Second, pre-initialize the matrix A to be N-by-N and contain zeros:

A = np.zeros((N, N), float)
Print("Zeros N by N", A)

The following should be printed on your screen:

Zeros N by N [[ 0.

3.

0.

[ 0.

0.

0.

0.

0.]

[ 0.

0.

0.

0.

0.]

[ 0.

0.

0.

0.

0.]

[ 0.

0.

0.

0.

0.]]

0.

0.

0.]

Third, fill the matrix A with N preceding price values for each value in b:
for i in range(N):
A[i, ] = c[-N - 1 - i: - 1 - i]
print("A", A)

[ 87 ]

Getting Familiar with Commonly Used Functions

Now, A looks like this:

A [[ 360.

4.

355.36

355.76

352.47

346.67]

[ 359.56

360.

355.36

355.76

352.47]

[ 352.12

359.56

360.

355.36

355.76]

[ 349.31

352.12

359.56

360.

355.36]

[ 353.21

349.31

352.12

359.56

360.

]]

The objective is to determine the coecients that satisfy our linear model by solving
the least squares problem. Employ the lstsq() function of the NumPy linalg
package to do this:
(x, residuals, rank, s) = np.linalg.lstsq(A, b)
print(x, residuals, rank, s)

The result is as follows:

[ 0.78111069 -1.44411737 1.63563225 -0.89905126 0.92009049]
[] 5 [ 1.77736601e+03
1.49622969e+01
8.75528492e+00
5.15099261e+00
1.75199608e+00]

The tuple returned contains the coecient x that we were after, an array comprising
residuals, the rank of matrix A, and the singular values of A.

5.

Once we have the coecients of our linear model, we can predict the next
price value. Compute the dot product (with the NumPy dot() function) of the
coecients and the last known N prices:
print(np.dot(b, x))

The dot product (see https://www.khanacademy.org/math/linearalgebra/vectors_and_spaces/dot_cross_products/v/vector-dotproduct-and-vector-length) is the linear combination of the coecients b
and the prices x. As a result, we get:
357.939161015

I looked it up; the actual close price of the next day was 353.56. So, our estimate with N = 5
was not that far o.

[ 88 ]

Chapter 3

What just happened?

We predicted tomorrow's stock price today. If this works in practice, we can retire early! See,
this book was a good investment, after all! We designed a linear model for the predictions.
The financial problem was reduced to a linear algebraic one. NumPy's linalg package has a
practical lstsq() function that helped us with the task at hand, estimating the coecients
of a linear model. After obtaining a solution, we plugged the numbers in the NumPy dot()
function that presented us an estimate through linear regression (see linearmodel.py):
from __future__ import print_function
import numpy as np
N = 5
c = np.loadtxt('data.csv', delimiter=',', usecols=(6,), unpack=True)
b = c[-N:]
b = b[::-1]
print("b", b)
A = np.zeros((N, N), float)
print("Zeros N by N", A)
for i in range(N):
A[i, ] = c[-N - 1 - i: - 1 - i]
print("A", A)

(x, residuals, rank, s) = np.linalg.lstsq(A, b)

print(x, residuals, rank, s)
print(np.dot(b, x))

Trend lines
A trend line is a line among a number of the so-called pivot points on a stock chart. As the
name suggests, the line's trend portrays the trend of the price development. In the past,
traders drew trend lines on paper but nowadays, we can let a computer draw it for us. In this
section, we shall use a very simple approach that probably won't be very useful in real life,
but should clarify the principle well.

[ 89 ]

Getting Familiar with Commonly Used Functions

Time for action drawing trend lines

Perform the following steps to draw trend lines:

1.

First, we need to determine the pivot points. We shall pretend they are equal to the
arithmetic mean of the high, low, and close price:
h, l, c = np.loadtxt('data.csv', delimiter=',', usecols=(4, 5,
6), unpack=True)
pivots = (h + l + c) / 3
print("Pivots", pivots)

From the pivots, we can deduce the so-called resistance and support levels.
The support level is the lowest level at which the price rebounds. The resistance
level is the highest level at which the price bounces back. These are not natural
phenomena, they are merely estimates. Based on these estimates, it is possible to
draw support and resistance trend lines. We will define the daily spread to be the
dierence of the high and low price.

2.

Define a function to fit line to data to a line where y = at + b. The function should
return a and b. This is another opportunity to apply the lstsq() function of the
NumPy linalg package. Rewrite the line equation to y = Ax, where A = [t 1]
and x = [a b]. Form A with the NumPy ones_like(), which creates an array,
where all the values are equal to 1, using an input array as a template for the
array dimensions:
def fit_line(t, y):
A = np.vstack([t, np.ones_like(t)]).T
return np.linalg.lstsq(A, y)[0]

3.

Assuming that support levels are one daily spread below the pivots, and that
resistance levels are one daily spread above the pivots, fit the support and
resistance trend lines:
t = np.arange(len(c))
sa, sb = fit_line(t, pivots - (h - l))
ra, rb = fit_line(t, pivots + (h - l))
support = sa * t + sb
resistance = ra * t + rb

[ 90 ]

Chapter 3

4.

At this juncture, we have all the necessary information to draw the trend lines;
however, it is wise to check how many points fall between the support and resistance
levels. Obviously, if only a small percentage of the data is between the trend lines,
then this setup is of no use to us. Make up a condition for points between the bands
and select with the where() function, based on the following condition:
condition = (c > support) & (c < resistance)
print("Condition", condition)
between_bands = np.where(condition)

These are the printed condition values:

Condition [False False
True False False

True

True

True

False False False True False False False

False False True True True False True]

True
True

True False False

True

True

True

Double-check the values:

print(support[between_bands])
print( c[between_bands])
print( resistance[between_bands])

The array returned by the where() function has rank 2, so call the ravel()
function before calling the len() function:
between_bands = len(np.ravel(between_bands))
print("Number points between bands", between_bands)
print("Ratio between bands", float(between_bands)/len(c))

You will get the following result:

Number points between bands 15
Ratio between bands 0.5

As an extra bonus, we gained a predictive model. Extrapolate the next day resistance
and support levels:
print("Tomorrows support", sa * (t[-1] + 1) + sb)
print("Tomorrows resistance", ra * (t[-1] + 1) + rb)

This results in the following output:

Tomorrows support 349.389157088
Tomorrows resistance 360.749340996

[ 91 ]

Getting Familiar with Commonly Used Functions

Another approach to figure out how many points are between the support and
resistance estimates is to use [] and intersect1d(). Define selection criteria
in the [] operator and intersect the results with the intersect1d() function:
a1 = c[c > support]
a2 = c[c < resistance]
print("Number of points between bands 2nd approach" ,len(np.
intersect1d(a1, a2)))

Not surprisingly, we get:

Number of points between bands 2nd approach 15

5.

Once more, plot the results:

plt.plot(t, c, label='Data')
plt.plot(t, support, '--', lw=2.0, label='Support')
plt.plot(t, resistance, '-.', lw=3.0, label='Resistance')
plt.title('Trend Lines')
plt.xlabel('Days')
plt.ylabel('Price ($)')
plt.grid()
plt.legend()
plt.show()

In the following plot, we have the price data and the corresponding support and
resistance lines:

[ 92 ]

Chapter 3

What just happened?

We drew trend lines without having to mess around with rulers, pencils, and paper charts.
We defined a function that can fit data to a line with the NumPy vstack(), ones_like(),
and lstsq() functions. We fit the data in order to define support and resistance trend lines.
Then, we figured out how many points are within the support and resistance range. We did
this using two separate methods that produced the same result.
The first method used the where() function with a Boolean condition. The second method
made use of the [] operator and the intersect1d() function. The intersect1d()
function returns an array of common elements from two arrays (see trendline.py):
from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt

def fit_line(t, y):

''' Fits t to a line y = at + b '''
A = np.vstack([t, np.ones_like(t)]).T
return np.linalg.lstsq(A, y)[0]
# Determine pivots
h, l, c = np.loadtxt('data.csv', delimiter=',', usecols=(4, 5, 6),
unpack=True)
pivots = (h + l + c) / 3
print("Pivots", pivots)
# Fit trend lines
t = np.arange(len(c))
sa, sb = fit_line(t, pivots - (h - l))
ra, rb = fit_line(t, pivots + (h - l))
support = sa * t + sb
resistance = ra * t + rb
condition = (c > support) & (c < resistance)
print("Condition", condition)
between_bands = np.where(condition)
print(support[between_bands])
print(c[between_bands])
print(resistance[between_bands])
between_bands = len(np.ravel(between_bands))

[ 93 ]

Getting Familiar with Commonly Used Functions

print("Number points between bands", between_bands)
print("Ratio between bands", float(between_bands)/len(c))
print("Tomorrows support", sa * (t[-1] + 1) + sb)
print("Tomorrows resistance", ra * (t[-1] + 1) + rb)
a1 = c[c > support]
a2 = c[c < resistance]
print("Number of points between bands 2nd approach" ,len(np.
intersect1d(a1, a2)))
# Plotting
plt.plot(t, c, label='Data')
plt.plot(t, support, '--', lw=2.0, label='Support')
plt.plot(t, resistance, '-.', lw=3.0, label='Resistance')
plt.title('Trend Lines')
plt.xlabel('Days')
plt.ylabel('Price ($)')
plt.grid()
plt.legend()
plt.show()

Methods of ndarray
The NumPy ndarray class has a lot of methods that work on the array. Most of the time,
these methods return an array. You may have noticed that many of the functions part of the
NumPy library have a counterpart with the same name and functionality in the ndarray
class. This is mostly due to the historical development of NumPy.
The list of ndarray methods is pretty long, so we cannot cover them all. The mean(),
var(), sum(), std(), argmax(), argmin(), and mean() functions that we saw earlier
are also ndarray methods.

Time for action clipping and compressing arrays

Here are a few examples of ndarray methods. Perform the following steps to clip and
compress arrays:

1.

The clip() method returns a clipped array, so that all values above a maximum
value are set to the maximum and values below a minimum are set to the minimum
value. Clip an array with values 0 to 4 to 1 and 2:
a = np.arange(5)
print("a =", a)
print("Clipped", a.clip(1, 2))
[ 94 ]

Chapter 3

This gives the following output:

a = [0 1 2 3 4]
Clipped [1 1 2 2 2]

2.

The ndarray compress() method returns an array based on a condition. For

instance, look at the following code:
a = np.arange(4)
print(a)
print("Compressed", a.compress(a > 2))

This returns the following output:

[0 1 2 3]
Compressed [3]

What just happened?

We created an array with values 0 to 3 and selected the last element with the compress()
function based on the a > 2 condition.

Factorial
Many programming books have an example of calculating the factorial. We should not break
with this tradition.

Time for action calculating the factorial

The ndarray class has the prod() method, which computes the product of the elements in
an array. Perform the following steps to calculate the factorial:

1.

Calculate the factorial of 8. To do this, generate an array with values 1 to 8 and call
the prod() function on it:
b = np.arange(1, 9)
print("b =", b)
print("Factorial", b.prod())

Check the result with your pocket calculator:

b = [1 2 3 4 5 6 7 8]
Factorial 40320

This is nice, but what if we want to know all the factorials from 1 to 8?

[ 95 ]

Getting Familiar with Commonly Used Functions

2.

No problem! Call the cumprod() method, which computes the cumulative product
of an array:
print("Factorials", b.cumprod())

It's pocket calculator time again:

Factorials [

24

120

720

5040 40320]

What just happened?

We used the prod() and cumprod() functions to calculate factorials (see
ndarraymethods.py):
from __future__ import print_function
import numpy as np
a = np.arange(5)
print("a =", a)
print("Clipped", a.clip(1, 2))
a = np.arange(4)
print(a)
print("Compressed", a.compress(a > 2))
b = np.arange(1, 9)
print("b =", b)
print("Factorial", b.prod())
print("Factorials", b.cumprod())

Missing values and Jackknife resampling

Data often misses values because of errors or technical issues. Even if we are not missing
values, we may have cause to suspect certain values. Once we doubt data values, derived
values such as the arithmetic mean, which we learned to calculate in this chapter, become
questionable too. It is common for these reasons to try to estimate how reliable the
arithmetic mean, variance, and standard deviation are.
A simple but eective method is called Jackknife resampling (see http://en.wikipedia.
org/wiki/Jackknife_resampling). The idea behind jackknife resampling is to
systematically generate datasets from the original dataset by leaving one value out at a time.
In eect, we are trying to establish what will happen if at least one of the values is wrong.
For each new generated dataset, we recalculate the arithmetic mean, variance, and standard
deviation. This gives us an idea of how much those values can vary.
[ 96 ]

Chapter 3

Time for action handling NaNs with the nanmean(), nanvar(),

and nanstd() functions
We will apply jackknife resampling to the stock data. Each value will be omitted by setting it
to Not a Number (NaN). The nanmean(), nanvar(), and nanstd() can then be used to
compute the arithmetic mean, variance, and standard deviation.

1.

First, initialize a 30-by-3 array for the estimates as follows:

estimates = np.zeros((len(c), 3))

2.

Loop through the values and generate a new dataset by setting one value to NaN at
each iteration of the loop. For each new set of values, compute the estimates:
for i in xrange(len(c)):
a = c.copy()
a[i] = np.nan
estimates[i,] = [np.nanmean(a), np.nanvar(a), np.nanstd(a)]

3.

Print the variance for each estimate (you can also print the mean or standard
deviation if you prefer):
print("Estimates variance", estimates.var(axis=0))

The following is printed on the screen:

Estimates variance [ 0.05960347

3.63062943

0.01868965]

What just happened?

We estimated the variances of the arithmetic mean, variance, and standard deviation of a
small dataset using jackknife resampling. This gives us an idea of how much the arithmetic
mean, variance, and standard deviation vary. The code for this example can be found in the
jackknife.py file in this book's code bundle:
from __future__ import print_function
import numpy as np
c = np.loadtxt('data.csv', delimiter=',', usecols=(6,), unpack=True)
# Initialize estimates array
estimates = np.zeros((len(c), 3))
for i in xrange(len(c)):

[ 97 ]

Getting Familiar with Commonly Used Functions

# Create a temporary copy and omit one value
a = c.copy()
a[i] = np.nan
# Compute estimates
estimates[i,] = [np.nanmean(a), np.nanvar(a), np.nanstd(a)]
print("Estimates variance", estimates.var(axis=0))

Summary
This chapter informed us about a great number of common NumPy functions. A few
common statistics functions were also mentioned.
After this tour through the common NumPy functions, we will continue covering
convenience NumPy functions such as polyfit(), sign(), and piecewise()
in the next chapter.

[ 98 ]

Get more information NumPy Beginner's Guide Third Edition

Where to buy this book

You can buy NumPy Beginner's Guide Third Edition from the Packt Publishing website.
Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet
book retailers.
Click here for ordering and shipping details.

www.PacktPub.com

Stay Connected: