Using Web Services

Chapter 13

Python for Informatics: Exploring Information

www.pythonlearn.com
 Data on the Web

• With the HTTP Request/Response well understood and well

supported, there was a natural move toward exchanging data
between programs using these protocols

• We needed to come up with an agreed way to represent data

going between applications and across networks

• There are two commonly used formats: XML and JSON

 Sending Data across the “Net”

Python Java
Dictionary HashMap

a.k.a. “Wire Protocol” - What we send on the

“wire”
 Agreeing on a “Wire Format”

<person>
<name>
De-Serialize
Chuck
Python </name> Java
Dictionary <phone> HashMap
Serializ 303 4456
e </phone>
</person>
XML
 Agreeing on a “Wire Format”

{
"name" : De-Serialize
Python "Chuck", Java
Dictionary "phone" : HashMap
Serializ "303-4456"
e
}

JSO
N
 XML “Elements” (or Nodes)
<people>
<person>
<name>Chuck</name>
• Simple Element
<phone>303 4456</phone>
</person>
• Complex Element <person>
<name>Noah</name>
<phone>622 7421</phone>
</person>
</people>
 XML
Marking up data to send across the network...

http://en.wikipedia.org/wiki/XML
 eXtensible Markup Language
• Primary purpose is to help information systems share structured
data

• It started as a simplified subset of the Standard Generalized

Markup Language (SGML), and is designed to be relatively
human-legible

http://en.wikipedia.org/wiki/XML
 XML Basics
• Start Tag <person>
<name>Chuck</name>
• End Tag
<phone type="intl">
• Text Content +1 734 303 4456
• Attribute
</phone>
<email hide="yes" />
• Self Closing Tag </person>
<person>

<name>Chuck</name>
White Space
<phone type="intl"> Line ends do not matter.
+1 734 303 4456 White space is generally
discarded on text elements.
</phone> We indent only to be
<email hide="yes" /> readable.
</person> <person>
<name>Chuck</name>
<phone type="intl">+1 734 303
4456</phone>
<email hide="yes" />
</person>
 Some XML...

http://en.wikipedia.org/wiki/XML
 XML Terminology

• Tags indicate the beginning and ending of elements

• Attributes - Keyword/value pairs on the opening tag of XML
• Serialize / De-Serialize - Convert data in one program into a
common format that can be stored and/or transmitted between
systems in a programming language-independent manner

http://en.wikipedia.org/wiki/Serialization
 XML as a Tree
<a> a
<b>X</b>
<c>
b c
<d>Y</d>
X d e
<e>Z</e>
</c>
</a>
Y Z
Elements Text
 XML Text and Attributes
a
<a>
<b w="5">X</b>
<c> w
b text
c
<d>Y</d> attrib node
<e>Z</e>
5 X d e
</c>
</a>
Y Z
Elements Text
 XML as a
Paths
<a> b c
<b>X</b>
<c>
X d e
<d>Y</d>

<e>Z</e> Y Z
</c> /a/b X
</a> /a/c/d Y
Elements Text /a/c/e Z
 XML Schema
Describing a “contract” as to what is acceptable XML.

http://en.wikipedia.org/wiki/Xml_schema
http://en.wikibooks.org/wiki/XML_Schema
 XML Schema
• Description of the legal format of an XML document

• Expressed in terms of constraints on the structure and content of

documents

• Often used to specify a “contract” between systems - “My system

will only accept XML that conforms to this particular Schema.”

• If a particular piece of XML meets the specification of the Schema

- it is said to “validate”
http://en.wikipedia.org/wiki/Xml_schema
 XML
Validation
XML
Document

Validator
XML Schema
Contract
 XML
XML
<person>
Document Validation
<lastname>Severance</lastname>
<age>17</age>
<dateborn>2001-04-17</dateborn>
</person>
XML Schema
Contract
<xs:complexType name=”person”>
Validator
<xs:sequence>
<xs:element name="lastname" type="xs:string"/>
<xs:element name="age" type="xs:integer"/>
<xs:element name="dateborn" type="xs:date"/>
</xs:sequence>
</xs:complexType>
Many XML Schema Languages
• Document Type Definition (DTD)

• http://en.wikipedia.org/wiki/Document_Type_Definition

• Standard Generalized Markup Language (ISO 8879:1986 SGML)

• http://en.wikipedia.org/wiki/SGML

• XML Schema from W3C - (XSD)

• http://en.wikipedia.org/wiki/XML_Schema_(W3C)
http://en.wikipedia.org/wiki/Xml_schema
 XSD XML Schema (W3C spec)
• We will focus on the World Wide Web Consortium (W3C) version

• It is often called “W3C Schema” because “Schema” is considered

generic

• More commonly it is called XSD because the file names end in

.xsd

http://www.w3.org/XML/Schema
http://en.wikipedia.org/wiki/XML_Schema_(W3C)
 XSD
Structure <person>
<lastname>Severance</lastname>
<age>17</age>
<dateborn>2001-04-17</dateborn>
</person>

• xs:element

• xs:sequence
<xs:complexType name=”person”>
<xs:sequence>

• xs:complexType
<xs:element name="lastname" type="xs:string"/>
<xs:element name="age" type="xs:integer"/>
<xs:element name="dateborn" type="xs:date"/>
</xs:sequence>
</xs:complexType>
<xs:element name="person">
<xs:complexType>
<xs:sequence>
XSD
<xs:element name="full_name" type="xs:string"
minOccurs="1" maxOccurs="1" /> Constraints
<xs:element name="child_name" type="xs:string"
minOccurs="0" maxOccurs="10" />
</xs:sequence>
</xs:complexType>
<person>
</xs:element>
<full_name>Tove Refsnes</full_name>
<child_name>Hege</child_name>
<child_name>Stale</child_name>
<child_name>Jim</child_name>
<child_name>Borge</child_name>
</person>

http://www.w3schools.com/Schema/schema_complex_indicators.as
p
 XSD
<xs:element name="customer" type="xs:string"/>
<xs:element name="start" type="xs:date"/>
Data
<xs:element name="startdate" type="xs:dateTime"/>
<xs:element name="prize" type="xs:decimal"/> Types
<xs:element name="weeks" type="xs:integer"/>

<customer>John Smith</customer>
It is common to represent
<start>2002-09-24</start>
time in UTC/GMT, given
that servers are often <startdate>2002-05-30T09:30:10Z</startdate>
scattered around the world. <prize>999.50</prize>
<weeks>30</weeks>

http://www.w3schools.com/Schema/schema_dtypes_numeric.asp
 ISO 8601 Date/Time Format

2002-05-30T09:30:10Z
Time of Timezone - typically
Year-month-day specified in UTC / GMT
day
rather than local time
zone.

http://en.wikipedia.org/wiki/ISO_8601
http://en.wikipedia.org/wiki/Coordinated_Universal_Time
http://www.w3schools.com/Schema/schema_example.asp
 xml1.py

import xml.etree.ElementTree as ET
data = '''<person>
<name>Chuck</name>
<phone type="intl">
+1 734 303 4456
</phone>
<email hide="yes"/>
</person>'''

tree = ET.fromstring(data)
print 'Name:',tree.find('name').text
print 'Attr:',tree.find('email').get('hide')
import xml.etree.ElementTree as ET
xml2.py
input = '''<stuff>
<users>
<user x="2">
<id>001</id>
<name>Chuck</name>
</user>
<user x="7">
<id>009</id>
<name>Brent</name>
</user>
</users>
</stuff>'''

stuff = ET.fromstring(input)
lst = stuff.findall('users/user')
print 'User count:', len(lst)
for item in lst:
print 'Name', item.find('name').text
print 'Id', item.find('id').text
print 'Attribute', item.get("x")
JavaScript Object Notation
 JavaScript Object Notation

• Douglas Crockford -
“Discovered” JSON
• Object literal notation in
JavaScript

http://www.youtube.com/watch?v=kc8BAR7SHJI
 json1.py
import json
data = '''{
"name" : "Chuck",
"phone" : {
"type" : "intl",
"number" : "+1 734 303 4456" JSON represents data
}, as nested “lists” and
"email" : { “dictionaries”
"hide" : "yes"
}
}'''

info = json.loads(data)
print 'Name:',info["name"]
print 'Hide:',info["email"]["hide"]
 json2.py
import json
input = '''[
{ "id" : "001",
"x" : "2",
"name" : "Chuck"
} ,
{ "id" : "009", JSON represents data
"x" : "7",
"name" : "Chuck"
as nested “lists” and
} “dictionaries”
]'''

info = json.loads(input)
print 'User count:', len(info)
for item in info:
print 'Name', item['name']
print 'Id', item['id']
print 'Attribute', item['x']
Service Oriented Approach

http://en.wikipedia.org/wiki/Service-oriented_architecture
 Service Oriented Approach
• Most non-trivial web applications use
services Application

• They use services from other applications

API
s
• Credit Card Charge

• Hotel Reservation systems

Servic
• Services publish the “rules” applications Servic
e
must follow to make use of the service e
(API)
 Multiple Systems

• Initially - two systems cooperate and

split the problem

• As the data/service becomes useful -

multiple applications want to use the
information / application

http://www.youtube.com/watch?v=mj-kCFzF0ME 5:15
 Web Services

http://en.wikipedia.org/wiki/Web_services
Application Program Interface
The API itself is largely abstract in that it specifies an
interface and controls the behavior of the objects
specified in that interface. The software that provides the
functionality described by an API is said to be an
“implementation” of the API. An API is typically defined
in terms of the programming language used to build an
application.

http://en.wikipedia.org/wiki/API
 Web Service Technologies
• SOAP - Simple Object Access Protocol (software)

• Remote programs/code which we use over the network

• Note: Dr. Chuck does not like SOAP because it is overly

complex

• REST - Representational State Transfer (resource focused)

• Remote resources which we create, read, update and delete

remotely http://en.wikipedia.org/wiki/SOAP_(protocol)
http://en.wikipedia.org/wiki/REST
https://developers.google.com/maps/documentation/geocoding/
{
"status": "OK", http://maps.googleapis.com/maps/api/geocode/json?s
"results": [ ensor=false&address=Ann+Arbor%2C+MI
{
"geometry": {
"location_type": "APPROXIMATE",
"location": {
"lat": 42.2808256,
"lng": -83.7430378
}
},
"address_components": [
{
"long_name": "Ann Arbor",
"types": [
"locality",
"political"
],
"short_name": "Ann Arbor"
}
],
"formatted_address": "Ann Arbor, MI, USA",
"types": [
"locality",
"political"
]
}
]
} geojson.py
import urllib
import json Enter location: Ann Arbor, MI
serviceurl = 'http://maps.googleapis.com/maps/api/geocode/json?' Retrieving http://maps.googleapis.com/...
Retrieved 1669 characters
while True: lat 42.2808256 lng -83.7430378
address = raw_input('Enter location: ')
if len(address) < 1 : break Ann Arbor, MI, USA
Enter location:
url = serviceurl + urllib.urlencode({'sensor':'false',
'address': address})
print 'Retrieving', url
uh = urllib.urlopen(url)
data = uh.read()
print 'Retrieved',len(data),'characters'

try: js = json.loads(str(data))
except: js = None
if 'status' not in js or js['status'] != 'OK':
print '==== Failure To Retrieve ===='
print data
continue

print json.dumps(js, indent=4)

lat = js["results"][0]["geometry"]["location"]["lat"]
lng = js["results"][0]["geometry"]["location"]["lng"]
print 'lat',lat,'lng',lng
location = js['results'][0]['formatted_address']
print location
geojson.py
 API Security and Rate Limiting
• The compute resources to run these APIs are not “free”

• The data provided by these APIs is usually valuable

• The data providers might limit the number of requests per day,
demand an API “key”, or even charge for usage

• They might change the rules as things progress...

 twitter2.py
import urllib
import twurl
import json
TWITTER_URL = 'https://api.twitter.com/1.1/friends/list.json'
while True:
print ''
acct = raw_input('Enter Twitter Account:')
if ( len(acct) < 1 ) : break
url = twurl.augment(TWITTER_URL,
{'screen_name': acct, 'count': '5'} )
print 'Retrieving', url
connection = urllib.urlopen(url)
data = connection.read()
headers = connection.info().dict
print 'Remaining', headers['x-rate-limit-remaining']
js = json.loads(data)
print json.dumps(js, indent=4)
for u in js['users'] :
print u['screen_name']
s = u['status']['text']
print ' ',s[:50]
Enter Twitter Account:drchuck
Retrieving https://api.twitter.com/1.1/friends ...
Remaining 14
twitter2.py
{
"users": [
{
"status": {
"text": "@jazzychad I just bought one .__.",
"created_at": "Fri Sep 20 08:36:34 +0000 2013",
},
"location": "San Francisco, California",
"screen_name": "leahculver",
"name": "Leah Culver",
},
{
"status": {
"text": "RT @WSJ: Big employers like Google ...",
"created_at": "Sat Sep 28 19:36:37 +0000 2013",
},
"location": "Victoria Canada",
"screen_name": "_valeriei",
"name": "Valerie Irvine",
],
}
Leahculver
@jazzychad I just bought one .__._
Valeriei
RT @WSJ: Big employers like Google, AT&amp;T are h
Ericbollens
RT @lukew: sneak peek: my LONG take on the good &a
halherzog
Learning Objects is 10. We had a cake with the LO,
def oauth() : hidden.py
return { "consumer_key" : "h7Lu...Ng",
"consumer_secret" : "dNKenAC3New...mmn7Q",
"token_key" : "10185562-ein2...P4GEQQOSGI",
"token_secret" : "H0ycCFemmwyf1...qoIpBo" }
 twurl.py
import urllib
import oauth
import hidden

def augment(url, parameters) :

secrets = hidden.oauth()
consumer = oauth.OAuthConsumer(secrets['consumer_key'], secrets['consumer_secret'])
token = oauth.OAuthToken(secrets['token_key'],secrets['token_secret'])
oauth_request = oauth.OAuthRequest.from_consumer_and_token(consumer,
token=token, http_method='GET', http_url=url, parameters=parameters)
oauth_request.sign_request(oauth.OAuthSignatureMethod_HMAC_SHA1(), consumer, token)
return oauth_request.to_url()

https://api.twitter.com/1.1/statuses/user_timeline.json?count=2
&oauth_version=1.0&oauth_token=101...SGI&screen_name=drchuck&oa
uth_nonce=09239679&oauth_timestamp=1380395644&oauth_signature=r
LK...BoD&oauth_consumer_key=h7Lu...GNg&oauth_signature_method=H
MAC-SHA1
 Summary
• Service Oriented Architecture - allows an application to be broken
into parts and distributed across a network

• An Application Program Interface (API) is a contract for interaction

• Web Services provide infrastructure for applications cooperating

(an API) over a network - SOAP and REST are two styles of web
services

• XML and JSON are serialization formats

 Acknowledgements / Contributions
Thes slide are Copyright 2010- Charles R. Severance (www.dr-
...
chuck.com) of the University of Michigan School of Information
and open.umich.edu and made available under a Creative
Commons Attribution 4.0 License. Please maintain this last slide
in all copies of the document to comply with the attribution
requirements of the license. If you make a change, feel free to
add your name and organization to the list of contributors on this
page as you republish the materials.

Initial Development: Charles Severance, University of Michigan

School of Information

… Insert new Contributors here