Creating Great REST and gRPC API Experiences (in Swift)

Creating Great
REST and gRPC
API Experiences
(in Swift)
Tim Burks
Google

October 4, 2000
CL 23852 by jeff
ProtocolBuffer, a class for encoding hierarchical data in
a compact binary form...
Co-designed and Rev. by Sanjay

10 billion+
API calls
every second

Protocol Buffers
a language-neutral, platform-neutral, extensible mechanism for serializing structured data.

“Protocol Buffers” means several things
1. A serialization mechanism
2. An interface description language
3. A methodology

A Serialization Mechanism
An encoded message is just a stream of bytes
[field_number<<3 + wire_type] [length if necessary] [data]...
$ hexdump /tmp/request.bin
0000000 0a 05 68 65 6c 6c 6f
0a is “0000 1010”, so
field_number = 1 and wire_type = 2

An Interface Deﬁnition Language
syntax = "proto3";
package echo;
// The message used for echo requests and responses.
message EchoMessage {
string text = 1;
}

A Methodology
% protoc echo.proto --swift_out=.
# This runs a plugin called protoc-gen-swift
#
# The plugin generates a Swift source file that implements
# the data structures defined in models.proto and code
# for reading and writing them as serialized bytes.

echo.pb.swift
// DO NOT EDIT.
//
// Generated by the Swift generator plugin for the protocol buffer compiler.
// Source: echo.proto
//
// For information on using the generated types, please see the documenation:
// https://github.com/apple/swift-protobuf/
...
/// The message used for echo requests and responses.
struct Echo_EchoMessage {
// SwiftProtobuf.Message conformance is added in an extension below. See the
// `Message` and `Message+*Additions` files in the SwiftProtobuf library for
// methods supported on all messages.
var text: String = String()
var unknownFields = SwiftProtobuf.UnknownStorage()
init() {}
}
// MARK: - Code below here is support for the SwiftProtobuf runtime.

A Data Deﬁnition Language
syntax = "proto3";
package models;
message User {
string id = 1;
string name = 2;
repeated PlantList plantLists = 3;
}
message PlantList {
string id = 1;
string name = 2;
repeated PlantListItem plants = 11;
}
message PlantListItem {
string id = 1;
string botanical_name = 2;
}

Protocol Buffers aren’t just for networking
class UserData {
static let sharedInstance = UserData()
public var user : Models_User
init() {
// read info from UserDefaults
if let userdata = UserDefaults.standard.data(forKey:"user") {
do {
user = try Models_User(serializedData:userdata)
} catch {
user = Models_User()
}
} else {
user = Models_User()
}
}
func save() {
DispatchQueue.main.async {
do {
try UserDefaults.standard.set(self.user.serializedData(), forKey:"user")
} catch (let error) {
print("SAVE ERROR (error)")
}
}
}

Interface Builder for Data
message Person {
string name = 1;
int32 id = 2;
string email = 3;
message PhoneNumber {
string number = 1;
}
repeated PhoneNumber phone = 4;
}
Interface Builder: Developers specify
their interfaces using a special tool,
tooling compiles and integrates that into
their apps.
Protocol Buffers: Developers specify
their data structures using a special
language, tooling compiles and
integrates that into their apps.

Let’s talk about Networked APIs
From the Google Cloud APIs Design Guide:
“Application Programming Interfaces that operate across a network of
computers. They communicate using network protocols including HTTP,
and are frequently produced by different organizations than the ones that
consume them.”

API Styles
1. Remote Procedure Call (RPC)
2. Representational State Transfer (REST)

An RPC is just a function call.
service Echo {
// Immediately returns an echo of a request.
rpc Get(EchoRequest) returns (EchoResponse) {}
}

What is a REST API?
From the Richardson Maturity Model (as described by Martin Fowler):
Level 3 Hypermedia Controls
Level 2 HTTP Verbs
Level 1 Resources
Level 0 HTTP

Hypermedia Controls???
HATEOAS (Hypertext As The Engine Of Application State)
<appointment>
<slot id = "1234" doctor = "mjones" start = "1400" end = "1450"/>
<patient id = "jsmith"/>
<link rel = "/linkrels/appointment/cancel" uri = "/slots/1234/appointment"/>
<link rel = "/linkrels/appointment/addTest" uri = "/slots/1234/appointment/tests"/>
<link rel = "self" uri = "/slots/1234/appointment"/>
<link rel = "/linkrels/appointment/changeTime" uri = "/doctors/mjones/slots?date=20100104@status=open"/>
<link rel = "/linkrels/appointment/updateContactInfo" uri = "/patients/jsmith/contactInfo"/>
<link rel = "/linkrels/help" uri = "/help/appointment"/>
</appointment>
(source: Martin Fowler)

REST
Requirements
- Roy Fielding
● A REST API should not be dependent on any single
communication protocol.
● A REST API should not contain any changes to the
communication protocols aside from filling-out or fixing the
details of underspecified bits of standard protocols.
● A REST API should spend almost all of its descriptive effort
in defining the media type(s) used for representing
resources and driving application state, or in defining
extended relation names and/or hypertext-enabled markup
for existing standard media types.
● A REST API must not define fixed resource names or
hierarchies (an obvious coupling of client and server).
● A REST API should never have “typed” resources that are
significant to the client.
● A REST API should be entered with no prior knowledge
beyond the initial URI (bookmark) and set of standardized
media types that are appropriate for the intended audience
(i.e., expected to be understood by any client that might use
the API).

1. HTTP/HTTPS
a. Paths describe resources (nouns)
b. HTTP verbs describe actions
2. JSON Payloads
REST in
Practice

OpenAPI
Industry standard format for describing for REST APIs
Originally designed for documentation, now with many other applications: API
Authoring, Validation, Documentation, Analysis, Search, Testing, Mocking,
Management, Code Generation

2010-11: Google API Discovery Format

Discovery Format Code Generators

github.com/google/google-api-go-client

github.com/google/google-api-python-client

github.com/google/google-api-swift-client

github.com/google/google-auth-library-swift

Authorization: How do we tell an API server
that it’s ok to respond to our requests?
We need to provide a token:
var request = URLRequest(url:url)
request.httpMethod = method
request.httpBody = ...
// add any needed request headers
request.setValue(authorization, forHTTPHeaderField:"Authorization")
How do we get that?

OAuth2 Authorization Flow
For an implementation in Swift, see BrowserTokenProvider.swift
Client Service
Authorization URL
Sign-In Page (HTML)
Human Confirmation
Browser Redirect w/ Code
Token Request (Code)
Token
Web
Browser

Easier ways to get tokens (1 of 2)
If you’re running inside a VM on Google Cloud Platform, you can get a token
from the Google Cloud Metadata Service.
% curl http://metadata/computeMetadata/v1/instance/service-accounts/default/token
{"access_token":"ya29.GqUBUgXcBmIt7vfHsWJT4qVzdhWxwEb2f3tamcA6ykrIsEANZfQnoH0HDCBnlCztLw
cD47w7YENghIucNUIIypLId4C5dXc4H8D93e17MrSbGRe4ipfoQhxPCIhIU3KJsvFjel0HcN2iwf8xURv2z1lWiN
2jkZjzLiMRWPKfSvtBVzuWkIo5uZ5u25IXle3tJ4SICh0-516sU84DFu0wkPO-q1xGpiff","expires_in":179
9,"token_type":"Bearer"}
Then pass “Bearer “ + access_token as the Authorization header.
See GoogleCloudMetadataTokenProvider.swift.

Easier ways to get tokens (2 of 2)
If you’re calling a Google API from anywhere, you can use a Service Account.
1. Create and download the account credentials.
2. Create a JWT and sign it with the account credentials.
3. POST the signed JWT to the Google Account Service and get a token!
See ServiceAccountTokenProvider.swift.

Aside: Build an HTTP server with swift-server/http

● REST APIs
○ Discovery Format
○ 250+API descriptions
○ 10+ generators
● gRPC APIs
○ Protocol Buffer Language
○ 40+ gRPC-based APIs
○ 7+ target languages

gRPC
Open-Source messaging system based on Google’s internal API architecture.
Used for code and documentation generation, API management, and support
services for APIs and microservices running at very large scale.

gRPC is owned by the Cloud Native Computing Foundation

gRPC Adoption
Microservices: in data centres
Streaming telemetry from network devices
Client Server communication/Internal APIs
Mobile Apps

gRPC
Transport Mechanism
Client → Server
Server → Client
Initial
Metadata
MsgMsg
End of
Stream
Status & Trailing
Metadata
Initial
Metadata
MsgMsg Msg
HTTP/2

Cross-Language Interoperability
Java
Service
Python
Service
GoLang
Service
C++
Service
gRPC
Service
gRPC
Stub
gRPC
Stub
gRPC
Stub
gRPC
Stub
gRPC
Service
gRPC
Service
gRPC
Service
gRPC
Stub

Build a Swift client
and service with
gRPC
https://github.com/grpc/gr
pc-swift/wiki/Try-gRPC-Swi
ft-in-a-Google-Compute-Eng
ine-VM
or just
https://goo.gl/ux4Txh

echo.proto
package echo;
service Echo {
// Immediately returns an echo of a request.
rpc Get(EchoRequest) returns (EchoResponse) {}
// Splits a request into words and returns each word in a stream of messages.
rpc Expand(EchoRequest) returns (stream EchoResponse) {}
// Collects a stream of messages and returns them concatenated when the caller closes.
rpc Collect(stream EchoRequest) returns (EchoResponse) {}
// Streams back messages as they are received in an input stream.
rpc Update(stream EchoRequest) returns (stream EchoResponse) {}
}
message EchoRequest {
// The text of a message to be echoed.
string text = 1;
}
message EchoResponse {
// The text of an echo response.
string text = 1;
}

main.swift (1/3)
// Unary
if client == "get" {
var requestMessage = Echo_EchoRequest()
requestMessage.text = message
print("Sending: " + requestMessage.text)
let responseMessage = try service.get(requestMessage)
print("get received: " + responseMessage.text)
}
// Server streaming
if client == "expand" {
requestMessage.text = message
let expandCall = try service.expand(requestMessage) {result in }
var running = true
while running {
do {
let responseMessage = try expandCall.receive()
print("Received: (responseMessage.text)")
} catch Echo_EchoClientError.endOfStream {
print("expand closed")
running = false
}
}
}

main.swift (2/3)
// Client streaming
if client == "collect" {
let collectCall = try service.collect() {result in }
let parts = message.components(separatedBy:" ")
for part in parts {
requestMessage.text = part
print("Sending: " + part)
try collectCall.send(requestMessage) {error in print(error)}
sleep(1)
}
let responseMessage = try collectCall.closeAndReceive()
}

main.swift (3/3)
// Bidirectional streaming
if client == "update" {
let updateCall = try service.update() {result in}
DispatchQueue.global().async {
var running = true
while running {
do {
let responseMessage = try updateCall.receive()
} catch Echo_EchoClientError.endOfStream {
print("update closed")
latch.signal()
break
print("error: (error)")
}
}
}
...
...
let parts = message.components(separatedBy:" ")
for part in parts {
requestMessage.text = part
try updateCall.send(requestMessage) {error in print(error)}
sleep(1)
}
try updateCall.closeSend()
// Wait for the call to complete.
latch.wait()
}
}

EchoService.swift (1/3)
class EchoProvider : Echo_EchoProvider {
// get returns requests as they were received.
func get(request : Echo_EchoRequest, session : Echo_EchoGetSession) throws -> Echo_EchoResponse {
var response = Echo_EchoResponse()
response.text = "Swift echo get: " + request.text
return response
}
// expand splits a request into words and returns each word in a separate message.
func expand(request : Echo_EchoRequest, session : Echo_EchoExpandSession) throws -> Void {
let parts = request.text.components(separatedBy: " ")
var i = 0
for part in parts {
response.text = "Swift echo expand ((i)): (part)"
try session.send(response)
i += 1
sleep(1)
}
}

// collect collects a sequence of messages and returns them concatenated when the caller closes.
func collect(session : Echo_EchoCollectSession) throws -> Void {
var parts : [String] = []
while true {
do {
let request = try session.receive()
parts.append(request.text)
} catch Echo_EchoServerError.endOfStream {
break
print("(error)")
}
}
response.text = "Swift echo collect: " + parts.joined(separator: " ")
try session.sendAndClose(response)
}

// update streams back messages as they are received in an input stream.
func update(session : Echo_EchoUpdateSession) throws -> Void {
var count = 0
while true {
do {
let request = try session.receive()
count += 1
response.text = "Swift echo update ((count)): (request.text)"
try session.send(response)
} catch Echo_EchoServerError.endOfStream {
break
print("(error)")
}
}
try session.close()
}
}

Why generate client libraries for APIs?
Make it easier to use Google APIs.
● Hide implementation details from API consumers.
● Develop API clients faster.
● Improve API client quality.
○ Better performance
○ Better consistency
○ Fewer bugs

Generated API Clients
a.k.a.
“GAPICs”

GAPIC Customers
Google Cloud APIs
Google Ads APIs
…other Google teams
…external API producers?

GAPIC Features and Beneﬁts
● Usability:
○ Trivial getting-started experience
○ Flattening
○ Natural paging
○ Long-running operations
○ Sample code for every method
○ Resource name types (e.g. for /projects/{project}/topics/{topic})
● Performance:
○ Batching
● Reliability:
○ Automatic retries

Trivial Getting-Started Experience
GAPIC client:
Channel channel =
NettyChannelBuilder.forAddress(API_ENTRY, API_PORT).build();
List<ClientInterceptor> interceptors = new ArrayList<>();
GoogleCredentials credentials =
GoogleCredentials.getApplicationDefault();
interceptors.add(
ChannelFactory.credentialInterceptor(credentials));
LoggingService stub =
LoggingServiceGrpc.newBlockingStub(channel, interceptors);
gRPC:
LoggingClient client = LoggingClient.create();

Automatic Retry and Flattening
GAPIC client:
LoggingService stub = ...;
// Note: this only does a simple retry, exponential backoff
// is more complex
DeleteLogRequest request = DeleteLogRequest.newBuilder()
.setLogName(LOG_NAME)
.build();
for (int i = 0; i < MAX_RETRY; i++) {
try {
stub.deleteLog(request);
} catch (RpcException e) {
if (i == MAX_RETRY) throw e;
}
}
gRPC:
client.deleteLog(LOG_NAME);

Natural Pagination
GAPIC client:
LoggingService stub = ...;
ListLogsRequest request = ListLogsRequest.newBuilder()
.setParentName("projects/" + PROJECT_ID)
.build();
while (true) {
ListLogsResponse response = stub.listLogs(request);
for (Log log : response.getLogsList()) {
System.out.printf("Log: %s%n", log.getDisplayName());
}
String nextPageToken = response.getNextPageToken();
if (nextPageToken != null) {
request = ListLogsRequest.newBuilder()
.setPageToken(nextPageToken).build();
} else {
break;
}
}
gRPC:
ParentNameOneOf parent =
ParentNameOneOf.from(ProjectName.create(PROJECT_ID));
for (Log log : client.listLogs(PARENT).iterateAll()) {
System.out.printf("Log: %s%n", log.getDisplayName());
}

Long-Running Operations
SpeechClient speechClient = SpeechClient.create();
OperationFuture<LongRunningRecognizeResponse> recognizeOperation =
speechClient.longRunningRecognizeAsync(config, audio);
…
LongRunningRecognizeResponse response = recognizeOperation.get();
● Java: OperationFuture
○ Polls the service until the LRO is done
○ Provides metadata as the LRO is in progress
○ Provides the final result

Resource Name Types
GAPIC client:
PublisherService stub = ...;
CreateTopicRequest request = CreateTopicRequest.newBuilder()
.setTopic("projects/my-project/topics/my-topic")
.build();
Topic response = stub.createTopic(request);
gRPC:
TopicAdminClient topicAdminClient = TopicAdminClient.create();
TopicName name = TopicName.create("my-project", "my-topic");
Topic response = topicAdminClient.createTopic(name);

GAPIC Project Architecture (hosted at github.com/googleapis)
gapic-generator-python
WANTED: ruby, swift, rust, elixir...
gapic-generator
(java, php, python, go,
csharp, node, ruby)
gapic-generator-kotlin
gapic-generator-go
gapic-generator-csharp
gapic-generator-typescript
One API
description
(Annotated
.proto ﬁles)
Many
language-
speciﬁc
client
libraries

Standard (Language-Independent) API Client Conﬁguration

“Showcase” API veriﬁcation

What can else we get from an API description format?
● Generated CLIs
● Generated Documentation
● API review and governance
● API management services
● Server frameworks and tools
● Generated mock servers
● Automatic testing
● API search and discovery

Manage Your Service With Endpoints
Develop, deploy and manage APIs on any Google Cloud Platform backend.
● User Authentication via JSON Web Token validation
● Logging and Monitoring
● API Keys
● Easy integration + deployment

Endpoints Architecture
GCE, GKE, Kubernetes or App Engine
Flexible
Environment Instance
GCE, GKE, Kubernetes or App Engine Flexible
Environment Instance (or off-GCP)
Extensible Service
Proxy Container
API Container
Google Service
Management API
User
Code
api
calls
gcloud
Config
deployment
Google Cloud
Console
Google Service
Control API
config Runtime
check & report
Load
balanci
ng
Stackdriver
Metrics &
logs
Metrics &
logs
visualized

Generate FileDescriptorSet
protoc kiosk.proto
-I api-common-protos
-I .
--include_imports
--include_source_info
--descriptor_set_out=generated/kiosk_descriptor.pb
--go_out=plugins=grpc:generated

Describing the Service
type: google.api.Service
config_version: 3
name: my-api.endpoints.my-gcp-project.cloud.goog
title: Kiosk gRPC API
apis:
- name: kiosk.Display

Starting Endpoints Proxy in Front of Application
gcloud endpoints services deploy service.yaml kiosk_descriptor.pb
./endpoints/START_ENDPOINTS_PROXY.sh my-gcp-project my-api <keyfile>

$ export KIOSK_PORT=8083
$ k list kiosks
FROM localhost:8083 LIST KIOSKS
rpc error: code = Unauthenticated
desc = Method doesn't allow
unregistered callers (callers
without established identity).
Please use API Key or other form of
API consumer identity to call this
API.
Starting Endpoints Proxy in Front of Application

$ export KIOSK_APIKEY=AIzaSy[...]bBo
$ k list kiosks
FROM localhost:8083 2018/09/21 19:08:25 Using API key: AIzaSy[...]bBo
LIST KIOSKS
id:1 name:"HTTP Kiosk" size:<width:1080 height:720 >
id:2 name:"HTTP Kiosk" size:<width:1080 height:720 >
Getting an API Key

Build a gRPC service, get a free REST API!

service Display {
rpc CreateKiosk(Kiosk) returns (Kiosk) {
option (google.api.http) = {post:"/v1/kiosks"};
}
rpc GetKiosk(GetKioskRequest) returns (Kiosk) {
option (google.api.http) =
{get: "/v1/kiosks/{id}" };
}
rpc DeleteKiosk(DeleteKioskRequest)
returns (google.protobuf.Empty) {
option (google.api.http) =
{ delete: "/v1/signs/{id}" };
}
[...]
}
Add REST annotations to your service
gRPC definitions with HTTP annotations
service Display {
rpc CreateKiosk(Kiosk) returns (Kiosk) {
}
rpc GetKiosk(GetKioskRequest) returns(Kiosk) {
}
rpc DeleteKiosk(DeleteKioskRequest)
returns (google.protobuf.Empty) {
}
[...]
}
gRPC definitions

# Add a kiosk
$ curl -d '{"name":"HTTP Kiosk", "size": { width: 1080, height: 720 } }'
localhost:8082/v1/kiosks?key=AIzaSy[...]bBo
# Get kiosk
$ curl localhost:8082/v1/kiosks/1?key=AIzaSy[...]bBo
Now you can use HTTP+JSON!

Use Endpoints, get a free developer portal!

Build GraphQL interfaces to gRPC services
with rejoiner

BONUS: Call Google RPC APIs with HTTP/1 using gRPC Fallback

We’re in the Summer of Code!

And don’t miss: Swift for Tensorﬂow

Creating Great REST and gRPC API Experiences (in Swift)

More Related Content

What's hot (20)

Similar to Creating Great REST and gRPC API Experiences (in Swift) (20)

More from Tim Burks (12)

Recently uploaded (20)

Creating Great REST and gRPC API Experiences (in Swift)