Failure is an Option: Scaling Resilient Feature Delivery

1
Failure is an Option:
Scaling Resilient Feature Delivery
Jamie Connelly
Sr. Product Manager
Mike Davis
Sr. Staff Engineer

● Customize the widgets on your page
to your preference
● This webinar is recorded and you will
receive the link with the slides in the
next few days
● We will have time for questions! Please
submit them in the Q&A box on the left
side of the screen
Housekeeping

3
Agenda
1. Why progressive delivery and experimentation?
2. How Optimizely helps you deliver more resilient features
3. Scaling resilience: Introducing Optimizely Agent
4. Benefits of deploying Optimizely as a microservice
5. Deep-dive into Optimizely Agent architecture
6. Dogfooding Case Study on Optimizely Agent
7. Q&A

“Our success at Amazon is a
function of how many
experiments we do per year,
per month, per week, per day.”
“Our company culture
encourages experimentation
and free flow of ideas.”
“We use experimentation
and testing to inform as
much of the business as we
can."
Culture of
experimentation
Ship every
11 Seconds
Ship every
22 seconds
Continuous
Code Deploys
Progressive Delivery and Experimentation
Greg Peters / Netflix CPO
Jeff Bezos / Amazon Founder
Larry Page / Google Founder
Ship every
3.5 Seconds

Am I building the right thing?
Can we quickly build it in the right way?
Can we catch bugs before our users do?
Will this change positively or negatively impact our KPI’s?
There is a guesswork gap
between your idea and getting it
into the hands of users
Ideation
Team
Guesswork Gap
Everyone

Define
& Design
Code
& Release
Ideation
Research
participants
Internal testersTeam
Iterative design & development
processes help, but don’t reflect
production environments or users
Guesswork Gap
Everyone

Define
& Design
Code
& Release
Ideation
Research
participants
Internal testersTeam
Iterative design & development
processes help, but don’t reflect
production environments or users
Guesswork Gap
Everyone$6B market cap loss
for Twitter in 1 day
in ad platform bugs

Define
& Design
Code
& Release
Measure &
Iterate
Optimize &
Grow
Ideation
Optimizely eliminates guesswork
across your entire lifecycle

Define
& Design
Code
& Release
Measure &
Iterate
Optimize &
Grow
Ideation
Rapid Prototyping &
Painted Door Tests
A/B/n tests
Feature Flagging
Phased & Targeted Rollouts
Omnichannel A/B/n testing
CRO & Messaging A/B/n Testing
Personalization
Our Progressive Delivery & Experimentation
Platform enables you to deliver better code,
better apps and growth faster

More frequent code depoys
208x
Prioritize experimentation
Elite product development teams have 208 times more
frequent code deployments and are twice as likely to
meet or exceed performance goals, like profitability.
Organizations that significantly outperform their
peers are almost twice as likely to prioritize
testing and experimentation.
Industry research confirms
2x

18
Scaling Resilient Feature Delivery

Key Feature Flagging Use Cases with Optimizely
Develop features concurrently by implementing
flags set to “off”. Deploy to master at any time
while avoiding merge conflicts.
Trunk-based development
Use a flag to send a small percentage of traffic
to a new build of your application to discovery
any unforeseen issues prior to rolling out to all
users.
Canary Releases
Give business users the power to grant
exclusive access to new features or beta
programs for your best customers or
prospects in a sales cycle.
Beta or Exclusive Access
With feature flags, you can quickly turn off
problem features without rolling back an entire
release.
Kill Switch
Deploy features behind “off” flags, then give
marketers or PMs the ability to turn on the
feature independent of a new deploy.
Dark Launches
Add parameterized variables to feature flags to
update features in real time, without a code
push or app update. Variables can also be
used to A/B test variations of features.
Remote Configuration

Optimize your search results, recommendations,
promotions, or any other algorithms to deliver the
most relevant content to your customers
Full Stack Experiment Use Cases
Algorithms
Test entirely new site or app designs before you
launch them, without any performance impact to
your customers
Front-End & Redesigns
Test features as you roll them out, to validate your
hypothesis and monitor your key business
metrics
Feature Experiments & Rollouts
Experiment with different prices, plans, or fees
personalized to your customer segments to drive
revenue
Price Testing
Don’t just test the customer experience: test
APIs, microservices, clusters, and architecture
designs to improve performance & reliability
Backend Performance
Test experiences on mobile devices, over-the-top
apps, IoT, conversational interfaces, and any
other experience that runs in code, and test
across devices.
Omnichannel

Rise of microservices
Monolith

Monolith Microservices

● Faster development
velocity
● Isolated services are more
resilient
● Purpose-built

50% of companies say that more than 50% of their new development utilizes microservices
-Source: 2018 O'Reilly The State of Microservices Maturity
● Faster development
velocity
● Isolated services are more
resilient
● Purpose-built

27
Alternative: Build a Service around SDK

28
Alternative: Build a Service around SDK
Drawback: Requires months of engineering time

Materials presented are for demonstration purposes only and may not be developed and / or delivered. Optimizely provides no guarantees and has no liability for information contained herein.
Introducing
Optimizely Agent
Scale feature delivery by implementing Optimizely
as a highly available, distributed microservice

Implementation: Monolith vs. SOA

SDK

SDK
SDK SDK
SDKSDK

34
At production scale,
customers maintain
1000s of services

36
Deployment Options
Deploy as a container Compile from source
> make run

37
Benefits of Using Optimizely Agent

38
Faster implementation time vs. building a service

39
Faster implementation time vs. building a service
● Get started in minutes
● Reach production scale in days vs.
months
● Customizable out-of-the-box service
● Open source, extend Agent to meet
custom needs

41
Maintenance and monitoring
100s to 1000s of
SDK instances
1 Service

42
Maintenance and monitoring
100s to 1000s of
SDK instances
1 Service
● Single location for
upgrades
● Fits microservices pattern
● Built-in admin API to track
health, application
performance

43
Governance and Onboarding
Team A Team B Team C
Optimizely Agent

44
Security
● One implementation for
security review
● Decreased network surface
area with less external network
requests
● Predictable outbound requests

45
Technical Deep dive on Optimizely Agent

46
Agent APIs
ADMIN APIs
● GET /health
● GET /metrics
● GET /info
● POST /webhooks/
CLIENT APIs
● POST /v1/activate
● POST /v1/track
● GET /v1/config
● POST /v1/override

48
Activate API
POST /v1/activate
Activate iterates over all qualifying features and
experiments and returns decisions for the given user.
Activate returns variables where applicable.
REQUEST BODY
● userId — string
● userAttributes — object
QUERY PARAMETERS
● featureKey — Array of strings
● experimentKey — Array of strings
● disableTracking — boolean
● type — enum array (“feature”, “experiment”)
● enabled — boolean
Example Request
{
"userId": "string",
"userAttributes": {
"country": "string",
"customer": false
}
}
Example Response
[
{
"featureKey": "string",
"experimentKey": "string",
"variationKey": "string",
"type": "feature",
"enabled": true,
"variables": {
"property1": "string",
"property2": "string"
}
}
]

49
Track API
POST /v1/track
Example Request
{
"userId": "string",
"userAttributes": {
"country": "string",
"customer": false
}
"eventTags": {
"revenue": int,
"product": "string"
}
}
Example Response
n/a
Track sends event and user details to Optimizely’s
analytics backend for experiment analysis.
REQUEST BODY
● userAttributes — object
● eventTags — object
QUERY PARAMETERS
● eventKey (required) — string

50
Config API
GET /v1/config
The config API allows you to access
the contents of your datafile — the
experiments and features required by
your application.
Example Response
{
"revision": "string",
"experimentMap": {
"experiment-key1": {
"id": 0,
"key": "string",
"variationsMap": {...}
},
"experiment-key2": {
"id": 0,
"key": "string",
"variationsMap": {...}
}
},
"featureMap": {
"feature-key1": {
"id": 0,
"key": "string",
"variablesMap": {...},
"experimentsMap": {...}
},
"feature-key2": {
"id": 0,
"key": "string",
"variablesMap": {...},
"experimentsMap": {...}
}
}
}

51
Override API
POST /v1/override
Override an experiment or feature test decision. The
override is stored in local memory. Should be used
debugging and testing purposes. Should not be
used for production overrides.
REQUEST BODY
● experimentKey — string
● variationKey — string
Example Request
{
"userId": "string",
"experimentKey": "string",
"variationKey": "string"
}
Example Request
n/a

52
Built for Enterprise Scale
● Native OAuth2 support
● Optional TLS encryption
● Network isolation between client and
admin APIs
● Support for multiple environments
● Open source software

54
Optimizely Dogfooding Case Study:
The Results Page

56
Optimizely Results API: SDK Implementation

57
= 70 servers

58
= 70 x10 = 700 Server Instances

59
= 700 SDK Instances

60
Optimizely Results API: Agent Implementation
= 12 clustered
Agent instances

61
Optimizely
Agent
Instance 1
Load Balancer
Optimizely
Agent
Instance N...
Customer
Service
/activate
Customer
Service
/track
Customer
Service
/activate
Optimizely CDN
Optimizely
Backend
datafiles
events

64
Agent and the Optimizely Results API
How it’s deployed:
● 12 containers
● 6 vCPUs
● 12GB RAM
What it’s doing:
● >700 clients
● 2.5K requests per second
● P95 latency* ~1ms per request
*All performance data depends on the connection speed, network, device, web page, and many other factors; actual results will vary.

65
How to get started with Agent

66
Recap on Optimizely Agent
1. Faster implementation in service-oriented architectures
2. Easier to maintain, monitor, and upgrade
3. Better governance and easy onboarding
4. Easier security and centralized networking

67
Getting Started with Agent
1. Check out the Github Repo today! https://github.com/optimizely/agent
2. Not using Optimizely? Get free feature flags with optimize.ly/agent-rollouts
3. Docs will be published next week
4. GA launch end of month

68
Q&A
Sign-up today for Optimizely Rollouts account:
optimize.ly/agent-rollouts

Failure is an Option: Scaling Resilient Feature Delivery

Recommended

More Related Content

What's hot (20)

Similar to Failure is an Option: Scaling Resilient Feature Delivery (20)

More from Optimizely (20)

Recently uploaded (20)

Failure is an Option: Scaling Resilient Feature Delivery