Top Performance Problems in Distributed Architectures

Top Performance Challenges in
Distributed Architectures

2
907Calls
41sec
97threads
104Calls
21sec
92threads
AWS CloudWatch API
Single Fetch
Bulk Fetch
$$$$
$$
Example #1: Building Monitoring for AWS
$0.01 / 1000 Calls

3
Click to edit Master title style
Lesson Learned: When moving to a more distributed architecture …
https://www.dynatrace.com/news/blog/monitoring-aws-fargate-with-dynatrace-testing-it-in-the-field/

4
…you also grow your dependencies …
https://www.dynatrace.com/news/blog/enterprise-cloud-ecs-microservices-and-dynatrace-at-neiman-marcus/

5
… and the potential impact of a failure grows!
https://www.dynatrace.com/news/blog/enterprise-cloud-ecs-microservices-and-dynatrace-at-neiman-marcus/
1 Bad Update
4 Impacted Services
Because of all
dependencies

6
Common Anti-Patterns we have seen
1. N+1 call
2. N+1 query
3. Payload flood
4. Granularity
5. Tight Coupling
6. Inefficient Service Flow
7. Timeouts, Retries, Backoff
8. Dependencies

8
N+1 Call Pattern
Monolithic Code
public double getQuote(String type) {
double quote=0;
for (Product product: products) {
quote += product.getValue();
}
return quote;
}
“Works” well within
a single process

9
N+1 Call Pattern
Product Service
Quote Service
1 call to Quote Service
=
44 calls to Product Service
1
14
17
13

10https://aws.amazon.com/solutions/case-studies/landbay/

11
X
Y
Z
Z
1
74
24
22 22
24 24
24 24
1
1 1
1
Subtotal: 243

12
N+1 Query Pattern
1 call to Quote Service
=
87 calls to DB
Quote Service
1
87
Product Service Product DB

13
Cascading N+1 Query Pattern
26k Database Calls
809
3956
4347
4773
3789
3915
4999

15
Payload Flood: Doc Creation Sequential across services

16
Payload Flood in Numbers: Full DOC sent between Services
18MB
20MB
21MB

17
Refactor: Only send relevant data to specialized services
69MB
31.6MB
vs

19
Granularity: Encryption carved out into separate service
Doc Processor Doc Transformer Doc Signer
Doc Encryption
Doc Shipment
Documents
316
1 1
2 6 6 6
118

21
Tightly coupled! Shall we really distribute/extract?
When “Breaking the Monolith” be aware …
1:1
https://www.dynatrace.com/news/blog/breaking-up-the-monolith-while-migrating-to-the-cloud-in-6-steps/

22
Inefficient Service Flow
drawing parallels to Web Performance Optimization

23
SFPO (Service Flow&Performance Optimization)
has to teach us how to optimize (micro)service
dependencies through Service Flows

24
Especially useful to identify: inefficient 3rd party services, recursive
call chains, N+1 Query Patterns, loading too much data, no data
caching, … -> sounds very familiar to WPO

25
Classical cascading effect of recursive
service calls!

26
Timeouts, Retries & Backoff
Credits go to Adrian Hornsby (@adhorn)

27
Bad Timeout & Retry Settings
From Adrian Hornsby (@adhorn): https://speakerdeck.com/adhorn/resiliency-and-availability-design-patterns-3742b5ba-e013-4f50-8512-00a65775f478?slide=31
App
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend = default (e.g: 60s)
INSERT
INSERT
Retry
Retry
Retry
User 1
ERROR: Failed to get connection from pool

28
Backoff between Retries
App
DB
Conn
Poool
INSERT
Timeout client side = 10s Timeout backend = 10s – time elapsed
Wait 2s before Retry
User 1
Backoff

29
Simple Exponential Backoff is not enough: Add Jitter
No jitter With jitter

31
Look beyond the “Tip of the Iceberg”:
Understanding Dependencies is critical!

32
Who is depending on me? What is the risk of change?

33
Recap - Common Anti-Patterns + Metrics to look at
1. N+1 call: # same Service Invocations per Request
2. N+1 query: # same SQL Invocations per Request
3. Payload flood: Transfer Size!
4. Granularity: # of Service Invocations across End-2-End Transaction
5. Tight Coupling: Ratio between Service Invocations
6. Inefficient Service Flow: # of Involved Services, # of Calls to each Service
7. Timeouts, Retries, Backoff: Pool Utilization, …
8. Dependencies: # of Incoming & Outcoming Dependencies

34
Automated Deployment Validation
https://github.com/keptn/pitometer
https://github.com/keptn, www.keptn.sh

35
Build #17
Build #21
Identify / Optimize Architectural Patterns
Recursive Calls, N+1 Call Pattern, Chatty Interfaces, No Caching Layer …
Automate Architectural Checks into CI/CD/CO!

36
Automate Performance Checks into CI/CD/CO!
How is Performance & Resource Consumption per Service Endpoint?
Build #17 Build #21

37
From Google: “Everything as Code” e.g: Enforce Architectural Rules

38
From Dynatrace: „Performance Signature as Code“ evaluated through Jenkins
“Performance Signature”
for Build Nov 16
for Build Nov 17
for every Build
“Multiple Metrics”
compared to prev
Timeframe
Simple Regression Detection
per Metric
https://www.neotys.com/performance-advisory-council/thomas_steinmaurer

39
Pitometer (part of @keptnProject): Metrics-based grading of a Deployment!
Metric Source &
Query
Grading Details
& Metric Score
Pitometer Specfile
Total Scoring
Objectives
2GB
Allocated Bytes (from Prometheus)
> 2GB: 0 Points
< 2GB: 20 Points
5%
2% < 2%: 0 Points
< 5%: 10 Points
> 5%: 20 Points
Conversion Rate (Dynatrace)
GraderSource
If value: 3GB
Score: 0
If value: 3.9%
Score: 10
Total Score: 10

40
Pitometer: Run Standalone - https://github.com/keptn/pitometer
Init
Source
Source
Grader
Run
Result

41
Pitometer in keptn
Autonomous Cloud Control Plane
prodstage
1: push
2: deploy (shadow)
3: test (performance)
4: evaluate (scalability KPIs)
6: deploy (blue/green)
7: evaluate (business KPIs)
8: operate (NoOps)
5: promote
Dev

42
Resources
• Keptn & Pitometer
• www.keptn.sh
• github.com/keptn
• github.com/keptn/pitometer
• Performance, Resiliency & Availablity Content
• Adrian Hornsby (AWS): https://speakerdeck.com/adhorn/resiliency-and-availability-design-patterns-
3742b5ba-e013-4f50-8512-00a65775f478
• Acacio Cruz (Google): https://www.spreaker.com/user/pureperformance/066-load-shedding-sre-at-
google-with-aca
• Thomas Steinmaurer (Dynatrace): https://www.neotys.com/performance-advisory-
council/thomas_steinmaurer

Top Performance Problems in Distributed Architectures

Recommended

More Related Content

What's hot (16)

Similar to Top Performance Problems in Distributed Architectures (20)

More from Andreas Grabner (14)

Recently uploaded (20)

Top Performance Problems in Distributed Architectures

Editor's Notes