Running Data Platforms Like Products

© Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0
Running Data Platforms
Like Products
June 14, 2018
Mike Koleno, CTO, Solstice, @_koleno
Dormain Drewitz, Sr Director, Pivotal Software, @DormainDrewitz

Cover w/ Image
Industry Context
From centralized DBA teams to
diffused database responsibilities…
From monolithic RDBMS to
distributed, NoSQL datastores...
From request/response paradigms
to event-driven architectures…
“Whatever the nascent
reconsideration and
reevaluation of database
technologies, processes
and procedures comes
to be called [...] change
is coming.”
- Steve O’Grady, RedMonk
http://redmonk.com/sogrady/2018/01/09/whither-the-dba/

What You Should Take Away from Today
Understanding the
product mindset for
your internal teams
(particularly data folks)
Get some ideas about
what your “read” data
products might be and
how to build them
Get some ideas about
what your “write” data
products might be and
how to build them

Part of This Complete Source of Long Lead Times
PMO
Analyst
Architect
Lead Developer
Development Team
Operations Team
QA
DBA
Task 1
Task 2
Task 3
Task 4
Task 6
Task 5
Task 7
Task 9
Task 8
Adapted from: https://content.pivotal.io/white-papers/crossing-the-value-stream-improving-development-with-pivotal-and-cloud-foundry

Part of This Complete Source of Long Lead Times
PMO
Analyst
Architect
Lead Developer
Development Team
Operations Team
QA
DBA
Task 1
Task 2
Task 3
Task 4
Task 6
Task 5
Task 7
Task 9
Task 8
DBA

Not one, but TWO ways to stop/delay delivery to prod

Data in SOAP-land
App1:
“I need
some
data”
App2:
“I need
some
data”
Mainframe
ESB

Data in SOAP-land
App1:
“I need
some
data”
App2:
“I need
some
data”
Mainframe
ESB
Problem A: The values don’t match because
App1 interpreted the data differently from App2

DBAs: Stewards of Extracts, Satellite Reports & Dashboards

Master Data Management Issues
App1
App2 DB from some
acquisition
DB
Contains user
email
addresses
Also contains
user email
addresses
User has
accounts in
both apps with
different email
addresses

Master Data Management Issues
App1
App2 DB from some
acquisition
DB
Contains user
email
addresses
Also contains
user email
addresses
User has
accounts in
both apps with
different email
addresses
Problem B: App3 comes along and doesn’t
know which email address to use

What if “data” was an internal
product?
Hypothesis

Cover w/ Image
Explain “internal product”?
● Clearly identified customer(s)
○ E.g. development teams,
infrastructure teams,
leadership
● Product pride
○ E. g. Branding, showcasing
customer success
https://content.pivotal.io/blog/product-all-the-things

Two Types of Data “Products”
Reads
As a developer, I need to use data.
To read-data product team: Make it
simple, easy, and consistent to get
the right data into my application.
Writes
As a developer, I need to write
data somewhere.
To write-data product team: Make
it easy for to do the right thing,
while keeping developers
productive

Yes, this aligns with CQRS
Source: http://www.kennybastani.com/2017/01/building-event-driven-microservices.html
“... use different models
to update and read
domain data”

Examples of data “products”: Reads

Massive data engineering abstracted for developers
Location data for over 100mn
places
(Black box)
Places API
Geocoding
API
Geolocation
API
Time Zone
API
Product team domain

HCSC: Rebuilding SOAP services
https://content.pivotal.io/geode-at-springone-platform-2017/high-performance-cloud-native-apis-using-apache-geode-anna-jung-paul-vermeulen

HCSC: Rebuilding SOAP services
https://content.pivotal.io/geode-at-springone-platform-2017/high-performance-cloud-native-apis-using-apache-geode-anna-jung-paul-vermeulen
Customer A
Product team domain
Customer B
Customer N

Example: Challenges with Stored Procedures
Today: A Data Free-For-All
● n applications = n ways of
getting data
● Everything requires a DBA
● No centralized data access
strategy, leading to duplicated
efforts, inconsistencies and
conflicts
● Business logic entombed in
monolithic applications’ stored
procedures
Future: Data Products
● n applications = 1 way of
getting data
● DBA rarely required
● Provides centralized point of
access to common sets of
enterprise data
● Business logic stored in API
endpoints in server-side code,
which is much easier to debug

Example: Challenges with Stored Procedures
Today: A Data Free-For-All
● n applications = n ways of
getting data
● Everything requires a DBA
● No centralized data access
strategy, leading to duplicated
efforts, inconsistencies and
conflicts
● Business logic entombed in
monolithic applications’ stored
procedures
Future: Data Products
● n applications = 1 way of
getting data
● DBA rarely required
● Provides centralized point of
access to common sets of
enterprise data
● Business logic stored in API
endpoints in server-side code,
which is much easier to debug
● Hard to debug
● Hard to add new consumers
● Heavy DBA burden
● Centralization/Rationalizatio
n of process
● CI/CD
● Run server-side apps on a
cloud-native platform
● Autonomy

Building the Data API Supermarket

Running Data Platforms Like Products

Anti-pattern: Sharing a database
● Autonomy to iterate
● Autonomy of tech stack
● Performance
● Scaling

Data APIs (Services)
● Microservices do not access data
layer directly
○ Except for those that implement
the data API
● A surface area to:
○ Implement access control
○ Implement throttling
○ Perform logging
○ Other policies…

Adding a Data API/Service layer
● Performance?
● Resilience?
● Polyglot persistence?
● Schema versioning?

Adding a Data API/Service layer - with cache
● Performance
○ In-memory, horizontally
scalable
● Resilience
○ Provides bulkhead
● Polyglot persistence?

Cover w/ Image
Every Microservice Needs a
Cache
https://www.youtube.com/watch?v=Rzdxgx3RC0Q&t=205s
Not just for performance
● Scaling
● Autonomy
● Resilience

Shift the system of record safely
● Performance
scalable
● Resilience
● Polyglot persistence
○ Strangler pattern to transfer
data ownership
Recommended reading:
http://www.kennybastani.com/2016/08/strangling-legacy-microservices-
spring-cloud.html#more

Data API evolve in support of the evolution of the
microservice(s)
● Performance
scalable
● Resilience
● Polyglot persistence
○ Strangler pattern to transfer
data ownership
● Schema versioning
○ Parallel deployments
V1
V2

ACH payments example: Multiple ways to move money

ACH payments example: Unifying payments via API product

Cover w/ Image
The Consultative DBA
● Easy access
○ NOT tickets
○ Email alias? Slack channel?
● Tooling
○ Own the path to production
● Workshoping
○ Start with the “why”
○ Listen
● Goals and Metrics
○ Metrics of effort
○ Extrinsic validation metrics

Running Data Platforms Like Products

Recommended

More Related Content

What's hot (20)

Similar to Running Data Platforms Like Products (20)

More from VMware Tanzu (20)

Recently uploaded (20)

Running Data Platforms Like Products