Python in Action (Part 1)

Python in Action
Presented at USENIX LISA Conference
November 16, 2007
David M. Beazley
http://www.dabeaz.com

(Part I - Introducing Python)

Copyright (C) 2007, http://www.dabeaz.com 1- 1

Course Overview

• Python Programming by example in two acts
• Part I : The Python Language
• Part II : Python Systems Programming
• "In Action" means doing useful things.


Prerequisites
• I'm going to assume that...
• you have written programs
• you know about basic data structures
• you know what a function is
• you know about basic system concepts
(ﬁles, I/O, processes, threads, network, etc.)
• I do not assume that you know Python

Target Audience

• This tutorial is aimed at programmers who
want to get some idea of what Python is all
about.
• I assume that you're interested in solving
practical problems.
• Tool building


My Background
• C/assembler programming
• Started using Python in 1996 as a control
language for physics software running on
supercomputers at Los Alamos.
• Author: "Python Essential Reference"
• Developer of several open-source packages
• Currently working on parsing/compiler
writing tools for Python.


What is Python?
• An interpreted, dynamically typed
programming language.
• In other words: A language that's similar to
Perl, Ruby, Tcl, and other so-called "scripting
languages."
• Created by Guido van Rossum around 1990.
• Named in honor of Monty Python

Why was Python Created?
"My original motivation for creating Python was
the perceived need for a higher level language
in the Amoeba [Operating Systems] project. I
realized that the development of system
administration utilities in C was taking too long.
Moreover, doing these things in the Bourne
shell wouldn't work for a variety of reasons. ...
So, there was a need for a language that would
bridge the gap between C and the shell."

- Guido van Rossum

Important Inﬂuences

• C (syntax, operators, etc.)
• ABC (syntax, core data types, simplicity)
• Unix ("Do one thing well")
• Shell programming (but not the syntax)
• Lisp, Haskell, and Smalltalk (later features)


Some Uses of Python
• Text processing/data processing
• Application scripting
• Systems administration/programming
• Internet programming
• Graphical user interfaces
• Testing
• Writing quick "throw-away" code

More than "Scripting"

• Although Python is often used for "scripting",
it is a general purpose programming language
• Major applications are written in Python
• Large companies you have heard of are using
hundreds of thousands of lines of Python.


Our Focus : Systems
• In this tutorial we will cover a slice of Python
• Language introduction
• Data processing/parsing
• Files and I/O
• Systems programming


Notable Omissions
• Object-oriented programming. Python fully
supports objects, but covering this would
require an entire class. Besides, it's not
needed to write useful programs.
• Web frameworks. There are a variety of
frameworks for building web sites and
Internet programming in Python. This too,
would require a dedicated class.


Getting Started


Where to get Python?
http://www.python.org

• Site for downloads, community links, etc.
• Current version: Python-2.5.1
• Supported on virtually all platforms


Support Files
• Program ﬁles, examples, and dataﬁles for this
tutorial are available here:
http://www.dabeaz.com/action

• Please go there and follow along


Running Python (Unix)
• Command line
shell % python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license"
>>>

• Integrated Development Environment (IDLE)

shell % idle or


Running Python (win)
• Start Menu (IDLE or PythonWin)


Python Interpreter
• All programs execute in an interpreter
• If you give it a ﬁlename, it interprets the
statements in that ﬁle in order
• Otherwise, you get an "interactive" mode
where you can experiment
• No separate compilation step


Interactive Mode
• Read-eval loop
>>> print "hello world"
hello world
>>> 37*42
1554
>>> for i in range(5):
... print i
...
0
1
2
3
4
>>>

• Executes simple statements typed in directly
• This is one of the most useful features

Creating Programs
• Programs are put in .py files
# helloworld.py
print "hello world"

• Source files are simple text files
• Create with your favorite editor (e.g., emacs)
• Note: There may be special editing modes
• There are many IDEs (too many to list)


Creating Programs
• Creating a new program in IDLE


Creating Programs
• Editing a new program in IDLE


Creating Programs
• Saving a new Program in IDLE


Running Programs
• In production environments, Python may be
run from command line or a script
• Command line (Unix)
shell % python helloworld.py
hello world
shell %

• Command shell (Windows)
C:Somewhere>c:python25python helloworld.py
hello world
C:Somewhere>


Running Programs (IDLE)
• Select "Run Module" (F5)

• Will see output in IDLE shell window

A Sample Program
• Dave's Mortgage
Dave has taken out a $500,000 mortgage from
Guido's Mortgage, Stock, and Viagra trading
corporation. He got an unbelievable rate of 4% and a
monthly payment of only $499. However, Guido, being
kind of soft-spoken, didn't tell Dave that after 2 years,
the rate changes to 9% and the monthly payment
becomes $3999.

• Question: How much does Dave pay and
how many months does it take?


mortgage.py
# mortgage.py

principle = 500000 # Initial principle
payment = 499 # Monthly payment
rate = 0.04 # The interest rate
total_paid = 0 # Total amount paid
months = 0 # Number of months

while principle > 0:
principle = principle*(1+rate/12) - payment
total_paid += payment
months += 1
if months == 24:
rate = 0.09
payment = 3999

print "Total paid", total_paid
print "Months", months


Python 101: Statements
# mortgage.py

principle
payment
=
=
500000
499
Each statement appears
#
#
Initial principle
Monthly payment
rate = 0.04 #The on its own line
interest rate

months += 1
if months == 24:
rate = 0.09 No semicolons
payment = 3999



Python 101: Comments
# mortgage.py


# starts a comment which
extends to the end of the line
months += 1
if months == 24:
rate = 0.09
payment = 3999



Python 101:Variables
# mortgage.py

principle = 500000 Variables are declared by
#Initial principle
payment = 499 #Monthly payment
rate = 0.04 assigning a name to a value.
#The interest rate
months = 0
• Same name rules as C
# Number of months

while principle > 0: ([a-zA-Z_][a-zA-Z0-9_]*)
months += 1
if months == 24: • You do not declare types
rate = 0.09
payment = 3999
like int, ﬂoat, string, etc.
print "Months", months • Type depends on value


Python 101: Keywords
# mortgage.py

payment = Python has a small set of
499 # Monthly payment
total_paid = keywords and statements
0 # Total amount paid

Keywords are payment
principle = principle*(1+rate/12) - C-like
months += 1and else import raise
assert
if months == 24: except in return
rate break
= 0.09 exec is try
class
payment = 3999 finally lambda while
continue for not yield
print "Total paid", total_paid from
def or
del
print "Months", months global pass
elif if print


Python 101: Looping
while executes a loop as
# mortgage.py

principle a condition is True
long as = 500000 # Initial principle
rate while expression:
= 0.04 # The interest rate
statements
months ...= 0 # Number of months

months += 1
if months == 24:
rate = 0.09
payment = 3999

loop body denoted
by indentation

Python 101: Conditionals
# mortgage.py
if-elif-else checks a condition
if expression:
statements
rate ... = 0.04 # The interest rate
elif expression:
statements
...
else:
statements = principle*(1+rate/12) - payment
principle
...
months += 1
if months == 24:
rate = 0.09
payment = 3999

body of conditional
denoted by indentation

Python 101: Indentation
# mortgage.py

rate = 0.04 : indicates that an indented
# The interest rate
total_paid
months
=
=
0
0
#
#
block will follow
Total amount paid
Number of months

months += 1
if months == 24:
rate = 0.09
payment = 3999



Python 101: Indentation
# mortgage.py


months += 1
if months == 24:
rate = 0.09
payment = 3999

Python"Total cares about consistent
print
only paid", total_paid
indentation in the same block

Python 101: Primitive Types
# mortgage.py

principle
payment
=
=
500000
499
Numbers:
# Initial principle
# Monthly payment
rate = 0.04 # • Integer
The interest rate
months = 0 # • Floating point
Number of months

months += 1
if months == 24:
rate = 0.09 Strings
payment = 3999



Python 101: Expressions
# mortgage.py

Python uses conventional
principle
payment
= 500000
= 499
# Initial principle
# Monthly payment
syntax for 0.04
rate = operators and
# The interest rate

monthsexpressions
total_paid
= 0
= 0
# Total amount paid
# Number of months

months += 1
if months == 24: Basic Operators
rate = 0.09
payment = 3999 + - * / // % ** << >> | & ^
< > <= >= == != and or not


Python 101: Output
# mortgage.py


print writes to standard output
• Items are separated by spaces

• Includes a terminating newline
months += 1
if months == 24:
• Works with any Python object
rate = 0.09
payment = 3999



Running the Program
• Command line
shell % python mortgage.py
Total paid 2623323
Months 677
shell %

• Keeping the interpreter alive (-i option or IDLE)
shell % python -i mortgage.py
Total paid 2623323
Months 677
>>> months/12
56
>>>

• In this latter mode, you can inspect variables
and continue to type statements.

Interlude
• If you know another language, you already
know a lot of Python
• Python uses standard conventions for
statement names, variable names,
numbers, strings, operators, etc.
• There is a standard set of primitive types
such as integers, ﬂoats, and strings that look
the same as in other languages.
• Indentation is most obvious "new" feature

Getting Help
• Online help is often available
• help() command (interactive mode)

• Documentation at http://www.python.org

dir() function
• dir() returns list of symbols
>>> import sys
>>> dir(sys)
['__displayhook__', '__doc__', '__excepthook__',
'__name__', '__stderr__', '__stdin__', '__stdout__',
'_current_frames', '_getframe', 'api_version', 'argv',
'builtin_module_names', 'byteorder', 'call_tracing',
'callstats', 'copyright', 'displayhook', 'exc_clear',
'exc_info', 'exc_type', 'excepthook', 'exec_prefix',
'executable', 'exit', 'getcheckinterval',
...
'version_info', 'warnoptions']

• Useful for exploring, inspecting objects, etc.


More on Relations
• Boolean expressions: and, or, not
if b >= a and b <= c:
print "b is between a and c"

if not (b < a or b > c):
print "b is still between a and c"

• Don't use &&, ||, and ! as in C
&& and
|| or
! not

• Relations do not require surrounding ( )

Line Continuation
• Line continuation for long statements ()
if product=="game" and type=="pirate memory"
and age >= 4 and age <= 8:
print "I'll take it!"

• Line continuation is not needed for any code
inside (), [], or { }
if (product=="game" and type=="pirate memory"
and age >= 4 and age <= 8):
print "I'll take it!"


More on Numbers
• Numeric Datatypes
a = True # A boolean (True or False)
b = 42 # An integer (32-bit signed)
c = 81237742123L # A long integer (arbitrary precision)
d = 3.14159 # Floating point (double precision)

• Integer operations that overﬂow become longs
>>> 3 ** 73
67585198634817523235520443624317923L
>>> a = 72883988882883812
>>> a
72883988882883812L
>>>

• Integer division truncates (for now)
>>> 5/4
1
>>>


More on Strings
• String literals use several quoting styles
a = "Yeah but no but yeah but..."

b = 'computer says no'

c = '''
Look into my eyes, look into my eyes,
the eyes, the eyes, the eyes,
not around the eyes,
don't look around the eyes,
look into my eyes, you're under.
'''

• Standard escape sequences work (e.g., 'n')
• Triple quotes capture all literal text enclosed

Basic String Manipulation
• Length of a string
n = len(s) # Number of characters in s

• String concatenation
s = "Hello"
t = "World"
a = s + t # a = "HelloWorld"

• Strings as arrays : s[n]
s = "Hello"
s[1]

s[1] 'e' H e l l o
s[-1] 'o' 0 1 2 3 4

• Slices : s[start:end]
s[1:3] "el"
s[1:3]

s[:4] "Hell" H e l l o
s[-4:] "ello" 0 1 2 3 4

Type Conversion
• Converting between data types
a = int(x) # Convert x to an integer
b = long(x) # Convert x to a long
c = float(x) # Convert x to a float
d = str(x) # Convert x to a string

• Examples:
>>> int(3.14)
3
>>> str(3.14)
'3.14'
>>> int("0xff")
255
>>>


Programming Problem
• Dave's stock scheme
After watching 87 straight
hours of "Guido's Insane
Money" on his Tivo, Dave
hatched a get rich scheme and
purchased a bunch of stocks.
He can no longer remember the evil scheme, but
he still has the list of stocks in a ﬁle "portfolio.dat".

• Write a program that reads this ﬁle, prints a
report, and computes how much Dave spent
during his late night stock "binge."


The Input File
• Input ﬁle: portfolio.dat
IBM 50 91.10
MSFT 200 51.23
GOOG 100 490.10
AAPL 50 118.22
YHOO 75 28.34
SCOX 500 2.14
RHT 60 23.45

• The data: Name, Shares, Price per Share


portfolio.py
# portfolio.py

total = 0.0
f = open("portfolio.dat","r")

for line in f:
fields = line.split()
name = fields[0]
shares = int(fields[1])
price = float(fields[2])
total += shares*price
print "%-10s %8d %10.2f" % (name,shares,price)

f.close()
print "Total", total


Python File I/O
# portfolio.py
"r" - Read
total = 0.0 "w" - Write
f = open("portfolio.dat","r") "a" - Append

for line in f:
fields = line.split() Files are modeled after C stdio.
name = fields[0]
shares = int(fields[1]) • f = open() - opens a ﬁle
• f.close() - closes the ﬁle
Data is just a sequence of bytes

f.close()


Reading from a File
# portfolio.py

total = 0.0
f Loops over all lines in the ﬁle.
= open("portfolio.dat","r")

for line in f: Each line is returned as a string.
name = fields[0]
Alternative reading methods:

f.close() • f.read([nbytes])
• f.readline()
• f.readlines()


String Processing
# portfolio.py
Strings have various "methods."
total = 0.0
f = split() splits a string into a list of strings
open("portfolio.dat","r")

for line in f:
name = fields[0]
line = 'IBM 50 91.10n'

f.close() fields = ['IBM', '50', '91.10']


Lists
# portfolio.py
A 'list' is an ordered sequence
open("portfolio.dat","r")It's like an array.
of objects.
total = 0.0
f =

for line in f:
name = fields[0]
fields = ['IBM', '50', '91.10']
f.close()


Types and Operators
# portfolio.py

total = 0.0
f To work with data, it must be

for line in f:
converted to an appropriate
type (e.g., number, string, etc.)
name = fields[0]

f.close() Operators only work if objects
have "compatible" types


String Formatting
# portfolio.py

total = 0.0
f % operator when applied to a

for string, formats it. Similar to
line in f:

= the C printf() function.
name fields[0]

f.close()
print "Total cost", total
format string values


Sample Output
shell % python portfolio.py
IBM 50 91.10
MSFT 200 51.23
GOOG 100 490.10
AAPL 50 118.22
YHOO 75 28.34
SCOX 500 2.14
RHT 60 23.45
Total 74324.5
shell %


More on Files
• Opening a ﬁle
f = open("filename","r") # Reading
g = open("filename","w") # Writing
h = open("filename","a") # Appending

• Reading f.read([nbytes]) # Read bytes
f.readline() # Read a line
f.readlines() # Read all lines into a list

• Writing g.write("Hello Worldn") # Write text
print >>g, "Hello World" # print redirection

• Closing
f.close()


More String Methods
s.endswith(suffix) # Check if string ends with suffix
s.find(t) # First occurrence of t in s
s.index(t) # First occurrence of t in s
s.isalpha() # Check if characters are alphabetic
s.isdigit() # Check if characters are numeric
s.islower() # Check if characters are lower-case
s.isupper() # Check if characters are upper-case
s.join(slist) # Joins lists using s as delimeter
s.lower() # Convert to lower case
s.replace(old,new) # Replace text
s.rfind(t) # Search for t from end of string
s.rindex(t) # Search for t from end of string
s.split([delim]) # Split string into list of substrings
s.startswith(prefix) # Check if string starts with prefix
s.strip() # Strip leading/trailing space
s.upper() # Convert to upper case


More on Lists
• A indexed sequence of arbitrary objects
fields = ['IBM','50','91.10']

• Can contain mixed types
fields = ['IBM',50, 91.10]

• Can contain other lists:
portfolio = [ ['IBM',50,91.10],
['MSFT',200,51.23],
['GOOG',100,490.10] ]


List Manipulation
• Accessing/changing items : s[n], s[n] = val
fields = [ 'IBM', 50, 91.10 ]

name = fields[0] # name = 'IBM'
price = fields[2] # price = 91.10
fields[1] = 75 # fields = ['IBM',75,91.10]

• Slicing : s[start:end], s[start:end] = t
vals = [0, 1, 2, 3, 4, 5, 6]
vals[0:4] [0, 1, 2, 3]
vals[-2:] [5, 6]
vals[:2] [0, 1]

vals[2:4] = ['a','b','c']
# vals = [0, 1, 'a', 'b', 'c', 4, 5, 6 ]


List Manipulation
• Length : len(s)
fields = [ 'IBM', 50, 91.10 ]
len(fields) 3

• Appending/inserting
fields.append('11/16/2007')
fields.insert(0,'Dave')

# fields = ['Dave', 'IBM', 50, 91.10, '11/16/2007']

• Deleting an item
del fields[0] # fields = ['IBM',50,91.10,'11/16/2007']


Some List Methods
s.append(x) # Append x to end of s
s.extend(t) # Add items in t to end of s
s.count(x) # Count occurences of x in s
s.index(x) # Return index of x in s
s.insert(i,x) # Insert x at index i
s.pop([i]) # Return element i and remove it
s.remove(x) # Remove first occurence of x
s.reverse() # Reverses items in list
s.sort() # Sort items in s in-place


Programming Problem
• Dave's stock portfolio
Dave still can't remember his evil "get rich
quick" scheme, but if it involves a Python
program, it will almost certainly involve some
data structures.

• Write a program that reads the stocks in
'portfolio.dat' into memory. Alphabetize the
stocks and print a report. Calculate the
initial value of the portfolio.


The Previous Program
# portfolio.py

total = 0.0
f = open("portfolio.dat","r")

for line in f:
name = fields[0]

f.close()


Simplifying the I/O
# portfolio.py

total = 0.0 Opens a ﬁle,
iterates over all lines,
for line in open("portfolio.dat"):
and closes at EOF.
name = fields[0]



Building a Data Structure
# portfolio.py

stocks = [] A list of "stocks"
name = fields[0]
holding= (name,shares,price)
Create a stock
stocks.append(holding) record and append
to the stock list
# print "Total", total


Tuples - Compound Data
# portfolio.py

stocks = [] A tuple is the most primitive
for compound data type (a sequence
line in open("portfolio.dat"):
fields = line.split() grouped together)
of objects
name = fields[0]
stocks.append(holding)

# print "Total", total How to write a tuple:
t = (x,y,z)
t = x,y,z # ()'s are optional
t = () # An empty tuple
t = (x,) # A 1-item tuple


A List of Tuples
# portfolio.py

stocks = [] stocks = [
('IBM', 50, 91.10),
('MSFT', 200, 51.23),
fields = line.split()('GOOG', 100, 490.10),
name = fields[0] ('AAPL', 50, 118.22),
('SCOX', 500, 2.14),
('RHT', 60, 23.45)
]

# print "Total", total This works like a 2D array
stocks[2] ('GOOG',100,490.10)
stocks[2][1] 100


Sorting a List
# portfolio.py

stocks = []

name = fields[0]
.sort() sorts
holding= (name,shares,price) a list "in-place"

stocks.sort() Note: Tuples are compared
element-by-element
('GOOG',100,490.10)
...
('AAPL',50,118.22)


Looping over Sequences
# portfolio.py

stocks = []

name = fields[0]
price = float(fields[2]) statement iterates over
for
any object that looks like a
sequence (list, tuple, ﬁle, etc.)
stocks.sort()
for s in stocks:
print "%-10s %8d %10.2f" % s



Formatted I/O (again)
# portfolio.py

stocks = []

name = fields[0]
stocks.append(holding) On each iteration, s is a tuple
(name,shares,price)
stocks.sort()
for s in stocks:
print "%-10s %8d %10.2f" % s

# print "Total cost", total
s = ('IBM',50,91.10)


Calculating a Total
# portfolio.py

stocks = []

name = fields[0]
Calculate the total
stocks.append(holding) value of the
stocks.sort() portfolio by summing shares*price
for s in stocks:
print "%-10s
across all of the stocks
%8d %10.2f" % s

total = sum([s[1]*s[2] for s in stocks])


Sequence Reductions
# portfolio.py

stocks = []

name = fields[0]
Useful functions for reducing data:
sum(s) - Sums items in a sequence
min(s) - Min value in a sequence
stocks.sort()
max(s) - Max value in a sequence
for s in stocks:
print "%-10s %8d %10.2f" % s



List Creation
# portfolio.py
stocks = [ [s[1]*s[2] for s in stocks] = [
stocks = []
('IBM',50,91.10), 50*91.10,
('MSFT',200,51.23), 200*51.23,
('GOOG',100,490.10), 100*490.10,
('AAPL',50,118.22), 50*118.22,
name = fields[0]
('SCOX',500,2.14), 500*2.14,
('RHT',60,23.45) 60*23.45
] price = float(fields[2]) ]

stocks.sort() This operation creates a new list.
for s in stocks:
print "%-10s %8d(known assa "list comprehension")
%10.2f" %



Finished Solution
# portfolio.py

stocks = []

name = fields[0]

stocks.sort()
for s in stocks:
print "%-10s %8d %10.2f" % s



Sample Output
shell % python portfolio.py
AAPL 50 118.22
GOOG 100 490.10
IBM 50 91.10
MSFT 200 51.23
RHT 60 23.45
SCOX 500 2.14
Total 72199.0
shell %


Interlude: List Processing
• Python is very adept at processing lists
• Any object can be placed in a list
• List comprehensions process list data
>>> x = [1, 2, 3, 4]
>>> a = [2*i for i in x]
>>> a
[2, 4, 6, 8]
>>>

• This is shorthand for this code:
a = []
for i in x:
a.append(2*i)


Interlude: List Filtering
• List comprehensions with a predicate
>>> x = [1, 2, -3, 4, -5]
>>> a = [2*i for i in x if i > 0]
>>> a
[2, 4, 8]
>>>

• This is shorthand for this code:
a = []
for i in x:
if i > 0:
a.append(2*i)


Interlude: List Comp.
• General form of list comprehensions
a = [expression for i in s
for j in t
...
if condition ]

• Which is shorthand for this:
a = []
for i in s:
for j in t:
...
if condition:
a.append(expression)


Historical Digression
• List comprehensions come from Haskell
a = [x*x for x in s if x > 0] # Python

a = [x*x | x <- s, x > 0] # Haskell

• And this is motivated by sets (from math)
a = { x2 | x ∈ s, x > 0 }

• But most Python programmers would
probably just view this as a "cool shortcut"


Big Idea: Being Declarative

• List comprehensions encourage a more
"declarative" style of programming when
processing sequences of data.
• Data can be manipulated by simply "declaring"
a series of statements that perform various
operations on it.


A Declarative Example
# portfolio.py

lines = open("portfolio.dat")
fields = [line.split() for line in lines]
stocks = [(f[0],int(f[1]),float(f[2])) for f in fields]

stocks.sort()
for s in stocks:
print "%-10s %8d %10.2f" % s



Files as a Sequence
# portfolio.py


stocks.sort()
for s in stocks: ﬁles are sequences of lines
print "%-10s %8d %10.2f" % s
'IBM 50 91.1n'
'MSFT 200 51.23n'
...


A List of Fields
# portfolio.py


This statement creates a list of string ﬁelds
stocks.sort()
for s in stocks:
print "%-10s %8d %10.2f" % s
'IBM 50 91.10n' [['IBM','50',91.10'],
'MSFT 200 51.23n' ['MSFT','200','51.23'],
...
print "Total", total ...
]


A List of Tuples
# portfolio.py


stocks.sort()
for s in stocks:
This creates a list of tuples with ﬁelds
print "%-10s %8d %10.2f" % s

converted to numeric values
[['IBM','50',91.10'],
print "Total", total [('IBM',50,91.10),
['MSFT','200','51.23'], ('MSFT',200,51.23),
... ...
] ]


Programming Problem
• "Show me the money!"
Dave wants to know if he can quit his day job and
join a band. The file 'prices.dat' has a list of stock
names and current share prices. Use it to find out.

• Write a program that reads Dave's portfolio,
the file of current stock prices, and
computes the gain/loss of his portfolio.
• (Oh yeah, and be "declarative")

Input Files
• portfolio.dat • prices.dat
IBM 50 91.10 IBM,117.88
MSFT 200 51.23 MSFT,28.48
GOOG 100 490.10 GE,38.75
AAPL 50 118.22 CAT,75.54
YHOO 75 28.34 GOOG,527.80
SCOX 500 2.14 AA,36.48
RHT 60 23.45 SCOX,0.63
RHT,19.56
AAPL,136.76
YHOO,24.10


Reading Data
# portvalue.py

# Read the stocks in Dave's portfolio

# Read the current stock prices
lines = open("prices.dat")
fields = [line.split(',') for line in lines]
prices = [(f[0],float(f[1])) for f in fields]

• This is using the same trick we just saw in
the last section


Data Structures
# portvalue.py



stocks = [ prices = [
('IBM',50,91.10), ('IBM',117.88),
('MSFT',200,51.23), ('MSFT',28.48),
... ('GE',38.75),
] ...
]


Some Calculations
# portvalue.py



initial_value = sum([s[1]*s[2] for s in stocks])
current_value = sum([s[1]*p[1] for s in stocks
for p in prices
if s[0] == p[0]])

print "Gain", current_value - initial_value


Some Calculations
# portvalue.py

fields = [line.split() for line prices = [
stocks = [ in lines]
('IBM',50,91.10), ('IBM',117.88),
('MSFT',200,51.23), ('MSFT',28.48),
# Read the current stock prices ('GE',38.75),
...
] ...
]

for p in prices
if s[0] == p[0]])



Some Calculations
# portvalue.py

('IBM',50,91.10), ('IBM',117.88),
('MSFT',200,51.23), ('MSFT',28.48),
...
] ...
]

for p in prices
if s[0] == p[0]])



Some Calculations
# portvalue.py

('IBM',50,91.10), ('IBM',117.88),
('MSFT',200,51.23), ('MSFT',28.48),
...
] ...
]

for p in prices
if s[0] == p[0]])

print "Gain", current-value - initial_value
Joining two lists on a common ﬁeld

Commentary
• The similarity between list comprehensions
and database queries in SQL is striking
• Both are operating on sequences of data
(items in a list, rows in a database table).
• If you are familiar with databases, list
processing operations in Python are
somewhat similar.


More on Tuples
• Tuples are commonly used to store records
(e.g., rows in a database)
t = ('IBM', 50, 91.10)

• You can access elements by index
t[0] 'IBM'
t[1] 50
t[2] 91.10

• You can also expand a tuple to variables
name, shares, price = t

name 'IBM'
shares 50
price 91.10


Tuples and Iteration
• Tuple expansion in for-loops
stocks = [('IBM', 50, 91.10),
('MSFT',200, 51.23),
...
]

total = 0.0
for name, shares, price in stocks:

• This can help clarify some code


Tuples and Iteration
• Example of code with tuple expansion
initial = sum([shares*price
for name, shares, price in stocks])

current = sum([s_shares*p_price
for s_name, s_shares, s_price in stocks
for p_name, p_price in prices
if s_name == p_name])

print "Gain", current - initial

Copyright (C) 2007, http://www.dabeaz.com 1-100

Iteration over multiple lists
• zip() function
names = ['IBM','AAPL','GOOG','YHOO','RHT']
shares = [50,50,100,20,60]

for name, nshares in zip(names,shares):
# name = 'IBM', nshares = 50
# name = 'AAPL',nshares = 50
# name = 'GOOG',nshares = 100
...

• zip() creates a list of tuples
shares = [50,50,100,20,60]

x = zip(names,shares)
# x = [('IBM',50),('AAPL',50),('GOOG',100),...]


Iteration with a counter
• enumerate() function
for i,n in enumerate(names):
# i = 0, n = 'IBM'
# i = 1, n = 'AAPL'
# ...

• Example: Reading a ﬁle with line numbers
for linenum,line in enumerate(open("filename")):
...


Programming Problem
• Dave's Hedge Fund
After an early morning coffee binge, Dave
remembers his 'get rich' scheme and hacks up a
quick Python program to automatically trade
stocks before leaving to go on his morning bike
ride. Upon return, he ﬁnds that his program has
made 1,000,000 stock purchases, but no trades!!

• Problem: Find out how many hours Dave will
have to work trimming hedges at $7/hour to
pay for all of these stocks.


The Input File
• Input ﬁle: bigportfolio.dat
AXP 30 62.38
BA 15 98.31
DD 30 50.60
CAT 10 77.99
AIG 5 71.26
UTX 5 69.71
HD 25 37.62
IBM 20 102.77
... continues for 1000098 total lines ...

• Total ﬁle size: 12534017 bytes (~12 MB)


hedge.py
# hedge.py

lines = open("bigportfolio.dat")

print "Hours of hedge clipping", total/7

• Output:
% python hedge.py
Total 1037156063.55
Hours of hedge trimming 148165151.936


Problem: Memory
• Our solution takes a LOT of memory

• The program is constructing several large lists


Temporary Lists
# hedge.py



Each of these operations
creates a new list of values.


hedge2.py (2nd Attempt)
# hedge2.py

total = 0.0
for line in open("bigportfolio.dat"):

print "Hours of hedge trimming", total/7.00

• This doesn't create any lists
• But we also lose the hip "declarative" style

An Observation
• Sometimes lists are constructed as a one-
time operation. Never to be used again!
# hedge.py

stocks = [(f[0],int(f[1]),float(f[2]) for f in fields]


• Notice in this code: data in ﬁelds, stocks, and
sum() is only used once.


Generated Sequences
• Generator expressions
x = [1,2,3,4]
y = (i*i for i in x)

• Creates an object that generates values
when iterating (which only works once)
>>> y
<generator object at 0x6e378>
>>> for a in y: print a
...
1
4
9
16
>>> for a in y: print a
...
>>>


hedge3.py (3rd Attempt)
# hedge3.py

fields = (line.split() for line in lines)
stocks = ((f[0],int(f[1]),float(f[2])) for f in fields)

total = sum(s[1]*s[2] for s in stocks)


A Generated Solution
# hedge3.py



Only a slight syntax change
lines = [line.split() for line in lines]

lines = (line.split() for line in lines)


A Generated Solution
# hedge3.py



For functions that operate on sequences, you
can generate the sequence in the function
argument (the syntax looks a little exotic).


Running the Solution
• It works!
shell % python hedge3.py
Total 1037156063.55
Hours of hedge trimming 148165151.936
shell %

• And it uses very little memory!

• And it runs about 3x faster than before

Interlude : Data Processing
• So far, we've used Python to process data
• And we used a lot of advanced machinery
• List comprehensions
• Generator Expressions
• Programming in a "declarative" style
• Question : Is Python an appropriate tool??
• What is the performance?

Python vs. Awk
• Let's put it head-to-head
{ total += $2 * $3 } END {
print "Hours of hedge trimming", total/7
}

• Performance (bigportfolio.dat)
AWK : 1.03 seconds
Python : 2.25 seconds

• Memory (bigportfolio.dat)
AWK : 516 KB
Python : 2560 KB

• System Notes: Mac Pro (2x2.66 Ghz Dual
Core Intel Xeon)

Commentary

• It's not surprising that Python is slower than
AWK. It's a much more complex language.

• However, it's not slow enough to make me
lose a lot of sleep about it.
• Your mileage may vary.


Segue: Ordered Data
• All examples have used "ordered" data
• Sequence of lines in a ﬁle
• Sequence of ﬁelds in a line
• Sequence of stocks in a portfolio
• What about unordered data?


Dictionaries
• A hash table or associative array
• Example: A table of stock prices
prices = {
'IBM' : 117.88,
'MSFT' : 28.48,
'GE' : 38.75,
'CAT' : 75.54,
'GOOG' : 527.80
}

• Allows random access using key names
>>> prices['GE'] # Lookup
38.75
>>> prices['GOOG'] = 528.50 # Assignment
>>>


Dictionaries
• Dictionaries as a data structure
• Named ﬁelds
stock = {
'name' : 'GOOG',
'shares' : 100,
'price' : 490.10
}

• Example use
>>> cost = stock['shares']*stock['price']
>>> cost
49010.0
>>>


Programming Problem
• "Show me the money!" - Part Deux
Dave wants to know if he can quit his day job and
join a band. The file 'prices.dat' has a list of stock
names and current share prices. Use it to find out.

• Write a program that reads Dave's portfolio,
the file of current stock prices, and
computes the gain/loss of his portfolio.
• Use dictionaries

Solution : Part I
• Creating a list of stocks in the portfolio
# portvalue2.py
# Compute the value of Dave's portfolio

stocks = []
record = {
'name' : fields[0],
'shares' : int(fields[1]),
'price' : float(fields[2])
}
stocks.append(record)


Dictionary Data Structures
# portvalue2.py
# Compute the value of Dave's portfolio

stocks = []
Each stock is a dict
record = {
'name' : fields[0], record = {
'shares' : int(fields[1]), 'name' : 'IBM',
'price' : float(fields[2]) 'shares' : 50
} 'price' : 91.10
stocks.append(record) }


Lists of Dictionaries
• A list of objects with "named ﬁelds."
# portvalue2.py stocks = [
{'name'
# Compute the value of Dave's portfolio :'IBM',
'shares' : 50,
stocks = [] 'price' : 91.10 },
for line in open("portfolio.dat"): {'name' :'MSFT',
fields = line.split() 'shares' : 200,
record = { 'price' : 51.23 },
'name' : fields[0], ...
]
'price' : float(fields[2])
} Example:
stocks.append(record) stocks[1] {'name' : 'MSFT',
'shares' : 200,
'price' : 51.23}

stocks[1]['shares'] 200


Solution : Part 2
• Creating a dictionary of current prices
prices = {}
for line in open("prices.dat"):
fields = line.split(',')
prices[fields[0]] = float(fields[1])

• Example:
prices {
'GE' : 38.75,
'AA' : 36.48,
'IBM' : 117.88,
'AAPL' : 136.76,
...
}


Solution : Part 3
• Calculating portfolio value and gain
initial = sum(s['shares']*s['price']
for s in stocks)

current = sum(s['shares']*prices[s['name']]
for s in stocks)

print "Current value", current

• Note: Using generator expressions


Solution : Part 3
• Calculating portfolio value and gain
initial = sum(s['shares']*s['price']
for s in stocks)

current = sum(s['shares']*prices[s['name']]
for s in stocks)

print "Current value", current
Fast price lookup
prices {
s = { 'GE' : 38.75,
'name' : 'IBM', 'AA' : 36.48,
'shares' : 50 'IBM' : 117.88,
'price' : 91.10 'AAPL' : 136.76,
} ...
}

More on Dictionaries
• Getting an item
x = prices['IBM']
y = prices.get('IBM',0.0) # w/default if not found

• Adding or modifying an item
prices['AAPL'] = 145.14

• Deleting an item
del prices['SCOX']

• Membership test (in operator)
if 'GOOG' in prices:
x = prices['GOOG']


More on Dictionaries
• Number of items in a dictionary
n = len(prices)

• Getting a list of all keys (unordered)
names = list(prices)
names = prices.keys()

• Getting a list of all values (unordered)
prices = prices.values()

• Getting a list of (key,value) tuples
data = prices.items()


The Story So Far

• Primitive data types: Integers, Floats, Strings
• Compound data: Tuples
• Sequence data: Lists
• Unordered data: Dictionaries


The Story So Far
• Powerful support for iteration
• Useful data processing primitives (list
comprehensions, generator expressions)
• Bottom line:
Signiﬁcant tasks can be accomplished
doing nothing more than manipulating
simple Python objects (lists, tuples, dicts)


Remaining Topics
• Details on Python object model
• Errors and exception handling
• Functions
• Modules
• Classes and objects


Object Mutability
• Objects fall into two categories
• Immutable (can't be changed)
• Mutable (can be changed)
• Mutable: Lists, Dictionaries
• Immutable: Numbers, strings, tuples
• All of this ties into memory management
(which is why we would care about such a
seemingly low-level implementation detail)

Copyright (C) 2007, http://www.dabeaz.com 2-
133

Variable Assignment

• Variables in Python are only names
• Assignment does not store a value into a
ﬁxed memory location (like C)
• It is only a name assignment to an object

134

Reference Counting
• Objects are reference counted
• Increased by assignment, inclusion
ref = 3
"a" 42
a = 42
b = a
"b"
c = [1,2]
c.append(b)
"c" [x, x, x]

• Can check using the is operator
>>> a is b
True
>>> a is c[2]
True

135

Reference Counting
• Important point: assignment does not copy!
ref = 1
a = 42 "a" 42

ref = 0
a = 37 "a" 42

ref = 1
37

• Creates a new object
• Makes the name refer to it
136

Reference Counting
• Common pitfall: “duplicating” a container
>>> a = [1,2,3,4]
>>> b = a "a"
[1,2,-10,4]
>>> b[2] = -10
"b"
>>> a
[1,2,-10,4]

• Other techniques must be used for copying
>>> a = [1,2,3,4]
>>> b = list(a) # Create a new list from a
>>> b[2] = -10
>>> a
[1,2,3,4]

137

Shallow Copies
• Creating a new list only makes a shallow copy
>>> a = [2,3,[100,101],4]
>>> b = list(a)
>>> a is b
False

• However, items in list copied by reference
>>> a[2].append(102) a
>>> b[2]
[100,101,102]
>>> 2 3 100 101 102 4

b

This list is
being shared

138

Deep Copying
• Makes a copy of an object and copies all
objects contained within it
• Use the copy module
>>> a = [2,3,[100,101],4]
>>> import copy
>>> b = copy.deepcopy(a)
>>> a[2].append(102)
>>> b[2]
[100,101]
>>>

139

Everything is an object
• Numbers, strings, lists, functions,
exceptions, classes, instances, etc...
• All objects are said to be "ﬁrst-class"
• Meaning: All objects that can be named can
be passed around as data, placed in
containers, etc., without any restrictions.
• There are no "special" kinds of objects

140

First-class example
• These functions do data conversions
int(x)
float(x)
str(x)

• Let's put them in a list
fieldtypes = [str, int, float]

• Let's make some tuples
fields = ['GOOG','100','490.10']
typed_fields = zip(fieldtypes,fields)
# [(str,'GOOG'),(int,'100'),(float,490.10)]

• Let's make values
values = [ty(field) for ty,field in typed_fields]
# values = ['GOOG',100,490.10]

141

First-class Commentary
• The fact that all objects are ﬁrst-class may
take some time to sink in.
• Especially if you come from C/C++
• Can be used for very compact, interesting
styles of programming.
• All named program elements can be treated
as data and used in surprising ways.

142

Object type
• All objects have a type>>> a = 42
>>> b = "Hello World"
>>> type(a)
<type 'int'>
>>> type(b)
<type 'str'>
>>>

• type() function will tell you what it is
• Typename usually a constructor function
>>> str(42)
'42'
>>>

143

Type Checking
• How to tell if an object is a speciﬁc type
if type(a) is list:
print "a is a list"

if isinstance(a,list): # Preferred
print "a is a list"

• Checking for one of many types
if isinstance(a,(list,tuple)):
print "a is a list or tuple"

144

Exceptions
• In Python, errors are reported as exceptions
• Causes the program to stop
• Example:
>>> prices = { 'IBM' : 91.10,
... 'GOOG' : 490.10 }
>>> prices['SCOX']
Traceback (most recent call last):
File "<stdin>", line 1, in ? Exception
KeyError: 'SCOX'
>>>


Builtin-Exceptions
• About two-dozen built-in exceptions
ArithmeticError
AssertionError
EnvironmentError
EOFError
ImportError
IndexError
KeyboardInterrupt
KeyError
MemoryError
NameError
ReferenceError
RuntimeError
SyntaxError
SystemError
TypeError
ValueError

• Consult reference
146

Exceptions

• Exceptions can be caught
• To catch, use try-except
try:
print prices["SCOX"]
except KeyError:
print "No such name"

• To raise an exception, use raise
raise RuntimeError("What a kerfuffle")


Exceptions
• Code can specify actions that must always run
f = open(filename,"r")
try:
...
finally:
f.close() # Runs regardless of exception

• ﬁnally block runs regardless of whether or not
an exception occurred
• Typically used to properly manage resources

Program Organization
• Python provides a few basic primitives for
structuring larger programs
• Functions
• Modules
• Classes
• Will use these as programs grow in size


Functions
• Deﬁned with the def statement
def read_portfolio(filename):
stocks = []
for line in open(filename):
record = { 'name' : fields[0],
'price' : float(fields[2]) }
stocks.append(record)
return stocks

• Using a function
stocks = read_portfolio('portfolio.dat')


Function Examples
# Read prices into a dictionary
def read_prices(filename):
prices = { }
return prices

# Calculate current value of a portfolio
def portfolio_value(stocks,prices):
return sum(s['shares']*prices[s['name']]
for s in stocks)


Function Examples
• A program that uses our functions
# Calculate the value of Dave's portfolio

stocks = read_portfolio("portfolio.dat")
prices = read_prices("prices.dat")
value = portfolio_value(stocks,prices)

print "Current value", value

• Commentary: There are no major surprises
with functions--they work like you would
expect.


Generator Functions
• A function that generates values (using yield)
• The primary use is with iteration (for-loop)
def make_fields(lines,delimeter=None):
for line in lines:
fields = line.split(delimeter)
yield fields

• Big idea: this function will generate a sequence
of values one at a time instead of returning
results all at once.
• Generation of values continues until return

Using a Generator Func
• Generator functions almost always used in
conjunction with the for statement
fields = make_fields(open("portfolio.dat"))

fields = make_fields(open("prices.dat"),',')
prices = {}
for f in fields:
prices[f[0]] = float(f[1])

• On each iteration of the for-loop, the yield
statement produces a new value. Looping
stops when the generator function returns


Modules

• As programs grow, you will want multiple
source ﬁles
• Also to re-use previous code
• Any Python source ﬁle is a module
• Just use the import statement


A Sample Module
# stockfunc.py

def read_portfolio(filename):
lines = open(filename)
fields = make_fields(lines)
return [ { 'name' : f[0],
'shares' : int(f[1]),
'price' : float(f[2]) } for f in fields]
# Read prices into a dictionary
def read_prices(filename):
prices = { }
return prices

# Calculate current value of a portfolio
def portfolio_value(stocks,prices):
return sum(s['shares']*prices[s['name']]
for s in stocks)


Using a Module
• importing a module
import stockfunc

stocks = stockfunc.read_portfolio("portfolio.dat")
prices = stockfunc.read_prices("prices.dat")
value = stockfunc.portfolio_value(stocks,prices)

• Modules deﬁne namespaces
• All contents accessed through module name


Modules as Namespaces
• All objects in your program always live inside
some module.
• For global variables, we're really talking about
variables at the module level.
# foo.py # bar.py
x = 42 x = "Hello World"

>>> import foo
These are different >>> import bar
>>> foo.x
42
>>> bar.x
'Hello World'
>>>

from module import
• Symbols from a module can be imported into
the namespace of another module
from stockfunc import read_portfolio


• Importing all symbols
from stockfunc import *

prices = read_prices("prices.dat")
value = portfolio_value(stocks,prices)

• This is only an export of symbol names. The
code in the imported module still runs in its
own module namespace however.

Python Standard Library
• Python comes with several hundred modules
• Text processing/parsing
• Files and I/O
• Systems programming
• Network programming
• Internet
• Standard data formats
• Will cover some of these in afternoon section

A Few Critical Modules
• Modules that are used quite frequently
• sys. Command line options, standard I/O
• math. Math functions (sqrt, sin, cos, etc.)
• copy. Copying of objects
• re. Regular expressions
• os. Operating system functions.

Classes and Objects
• Python provides full support for objects
• Deﬁned with the class statement
class Stock(object):
def __init__(self,name,shares,price):
self.name = name
self.shares = shares
self.price = price
def value(self):
return self.shares * self.price
def sell(self,nshares):
self.shares -= nshares


Using an Object
• Creating an object and calling methods
>>> s = Stock('GOOG',100,490.10)
>>> s.name
'GOOG'
>>> s.shares
100
>>> s.value()
49010.0
>>> s.sell(25)
>>> s.shares
75

• Basically, an object is just a way to package data
and functions together.


Classes and Methods
• A class is a just a collection of "methods"
• A method is just a function
self.name = name
self.price = price
methods def value(self):


Methods and Instances
• Methods always operate on an "instance"
• Passed as the ﬁrst argument (self)
self.name = name
self.price = price instance
def value(self):


Creating Instances
• Class used as a function to create instances
• This calls __init__() (Initializer)
self.name = name
self.price = price
def value(self):
>>> s = Stock('GOOG',100,490.10)
>>> print s
<__main__.Stock object at 0x6b910>
>>>


Instance Data
• Each instance holds data (state)
• Created by assigning attributes on self
self.name = name
self.shares = shares Instance data
self.price = price
def value(self):
>>> s = Stock('GOOG',100,490.10)
>>> s.name
'GOOG'
>>> s.shares
100
>>>


Calling Methods
• Methods are invoked on an instance
• Instance is passed as ﬁrst parameter
self.name = name
self.price = price
def value(self):
>>> s = Stock('GOOG',100,490.10)
>>> s.value()
49010.0
>>> s.sell(50)
>>>


Object Commentary
• There is much more to objects in Python
• However, I made a conscious decision not
to make objects the primary focus of this
tutorial.
• We will use some simple classes later, but I
won't be going to more detail on how
classes work or some of their more
advanced features.


Historical Note
• Classes were one of the last features added to
Python when it was ﬁrst created (almost as an
afterthought).
• Although knowing about classes is important,
Python does not treat them as a cultish religion
or the one true path to enlightenment.
• You can write very powerful programs without
using classes much at all (will see later)


The End of the Intro
• Python has a small set of very useful datatypes
(numbers, strings, tuples, lists, and dictionaries)
• There are very powerful operations for
manipulating data
• Programs can be organized using functions,
modules, and classes
• This is the essential information you need to
know to get started.


Python in Action (Part 1)

Recommended

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to Python in Action (Part 1) (20)

Recently uploaded (20)

Python in Action (Part 1)