SlideShare a Scribd company logo
Modeling the Performance of
Ultra-Large-Scale Systems
1
Using
Layered Simulations
[Slide courtesy of Randy Shoup, eBay Distinguished Architect] 2
ULS are Everywhere, and they’re HUGE!
1. Performance Testing should happen …
4
development
process
here?
here!
2. One Model to Rule Them All?
1. How do response time differ
between these two mechanism?
5
3. Different Stakeholders, Different Interests
End User Developer
System
Engineer
System
Integrator
Sales
Rep
6
An Analogy: The “4+1” View Model
Scenario
Logical View
Process
View
Development
View
Physical
View
8
The “3+1” View Model for
Performance Modeling
Scenario
World
View
Physical
View
Component
View
9
Component
Layer
A Layered Simulation Model!
10
World
View
Layer
Physical
Layer
Case Study 1
Bottleneck Detection in RSS Cloud
• RSS blogging service providers (e.g., Wordpress)
• Evaluation of two notification delivery mechanisms:
– Periodic Pull
– Real-time Push
Q1. How does response time differ?
Q2. How does resource usage
(e.g., CPU, memory) look like?
11
Component
Layer
The RSS Cloud Model
12
World
View
Layer
Physical
Layer
CPU is the Bottleneck!
13
Case Study 2
Evaluating a Distributed Monitor
1. How do response time differ
between these two mechanism?
14
Configuration Score
16
DISCRETIZE
fuzzy values
LESS is MORE
Distributed vs. Centralized
17
0.1Hz
0.2Hz
0.3Hz
0.1Hz
0.2Hz
0.3Hz
faster “cheaper”
lower resource usage
Keeping the Model Up-to-Date
• (What-if) Experimenting with different
configuration/deployment scenarios
• (Digging deeper) Adding new “layers”, as system is
understood better
• (Design evolution) Updating the simulation models as
the system evolves
18
GH'@&+/) +A 12(&'I &-829'-3) *#%'3144&2'J '
6!
1) +) 789: ) 02!
9(8; ) <<!
3&+&K'
3&+&F'
LH'7$M&+&2,'",1N&3) #%&+-5'7$M&+&2,'O2,&+&-,-'
?2%'= -&+' 7&B&#) 4&+'
". -,&A ''
?29$2&&+'
". -,&A ''
O2,&9+1,) +'
"1#&-''
0&4'
=!
Component
Layer'
: 'Z1. &+&%'"$A *#18) 2'P ) %&#F'
"I !
World
View
Layer'
Physical
Layer'
7$-,+$S*,&%'B-H'&2,+1#$d&%!
"F!
eHG<d'
eHL<d'
eHQ<d'
TABLE IX
SIMULATION RESULTS FOR THE ORIGINAL DESIGN OF THE PERFORMANCE MONITOR.
DataCollection Response time(s) Cost per Central Monitor Central Monitor Central Monitor
Frequency (Hz) transmission ($) Thread Util. (%) CPU Util. (%) RAM Util. (%)
0.1 0.2 2.2 31.0 42.6 37.2
0.2 0.3 2.7 43.7 68.2 47.6
0.3 0.4 3.1 60.2 86.6 59.2
the CPU of the central monitor is close to running at its full
capacity, which will likely result in system instability. More-
over, whiletheoriginal designhaslower cost per transmission,
be validated as information becomes available throug
development. Due to the lack of access to production
we could only estimate a list of resources and proce
eHG<d'
eHL<d'
eHQ<d'
/1-,&+! U(3&14&+X!
#) D&+'+&-) *+(&'*-19&!
QUESTIONS!
20
Backup Slides
21
FOUR CHALLENGES
The Current Practice of Performance Verification
22
1. Too Late in the Development Lifecycle
• Design changes are not evaluated until after
code is written
– Happens at the last stage of a delayed schedule
23
Requirements
Design
Implementation
Verification
Maintenance
2. Lots of Data
• Industrial case studies
have 2000> counters
• Time consuming to
analyze
• Hard to compare more
than 2 tests at once
24
3. No Documented Behavior
25
• Analysts have different perceptions of
performance regressions
• Analysis may be influenced by
– Analyst’s knowledge
– Deadline
4. Heterogeneous Environments
• Multiple labs to parallelize test executions
– Hardware and software may differ
– Tests from one lab may not be used to analyze
tests from another lab
26
Categorize Each Challenge
27
AT THE DESIGN LEVEL
Performance Verification
28
Evaluate Design Changes through
Performance Modeling
• Analytical models are often not suitable for all
stakeholders
– Abstract mathematical and statistical concepts
• Simulation models can be implemented with
support of existing framework
– Visualization
– No systematic approach to construct models that
can be used by different stakeholders
29
Layered Simulation Model
30
Physical layerComponent layer
World view layer
Can the current
infrastructure support
the projected growth of
users?
Investigate
threading model
Hardware resource
utilization
Case Studies
• We conducted two case studies
– RSS Cloud
• Show the process of constructed the model
• Derive the bottleneck of the application
– Performance monitor for ULS systems
• Evaluate whether or not an organization should re-
architect the software
• Our model can be used to extract important
information and aid in decision making
31
Ad

Recommended

Automated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise Applications
SAIL_QU
 
Animated Visualization of Software History Using Software Evolution Storyboards
Animated Visualization of Software History Using Software Evolution Storyboards
SAIL_QU
 
Detecting Interaction Coupling from Task Interaction Histories
Detecting Interaction Coupling from Task Interaction Histories
SAIL_QU
 
Understanding the Rationale for Updating a Function's Comment
Understanding the Rationale for Updating a Function's Comment
SAIL_QU
 
Log Engineering: Towards Systematic Log Mining to Support the Development of ...
Log Engineering: Towards Systematic Log Mining to Support the Development of ...
SAIL_QU
 
Empircal Studies of Performance Bugs & Performance Analysis Approaches for La...
Empircal Studies of Performance Bugs & Performance Analysis Approaches for La...
SAIL_QU
 
Mining Performance Regression Testing Repositories for Automated Performance ...
Mining Performance Regression Testing Repositories for Automated Performance ...
SAIL_QU
 
Supporting Software Evolution Using Adaptive Change Propagation
Supporting Software Evolution Using Adaptive Change Propagation
SAIL_QU
 
Software engineering- system testing
Software engineering- system testing
Tejas Mhaske
 
Testing Technique
Testing Technique
Ajeng Savitri
 
Documented Requirements are not Useless After All!
Documented Requirements are not Useless After All!
Lionel Briand
 
Testing Plan
Testing Plan
Ajeng Savitri
 
Testing throughout the software life cycle
Testing throughout the software life cycle
Alfa Rizki Harahap
 
Automated Change Impact Analysis between SysML Models of Requirements and Design
Automated Change Impact Analysis between SysML Models of Requirements and Design
Lionel Briand
 
Defect Prediction Over Software Life Cycle in Automotive Domain
Defect Prediction Over Software Life Cycle in Automotive Domain
RAKESH RANA
 
Survey on Software Defect Prediction
Survey on Software Defect Prediction
Sung Kim
 
naveed-kamran-software-architecture-agile
naveed-kamran-software-architecture-agile
Naveed Kamran
 
System development phases
System development phases
PrabinBista05
 
Chaper 1 sdlc
Chaper 1 sdlc
Azrul Aziz
 
Fundamentals of Software Engineering
Fundamentals of Software Engineering
Madhar Khan Pathan
 
Survey on Software Defect Prediction (PhD Qualifying Examination Presentation)
Survey on Software Defect Prediction (PhD Qualifying Examination Presentation)
lifove
 
Survey on Software Defect Prediction
Survey on Software Defect Prediction
lifove
 
Maintenance, Re-engineering &Reverse Engineering in Software Engineering
Maintenance, Re-engineering &Reverse Engineering in Software Engineering
Manish Kumar
 
Software engineering 21 integration testing
Software engineering 21 integration testing
Vaibhav Khanna
 
Neotys PAC - Bruno Da Silva
Neotys PAC - Bruno Da Silva
Neotys_Partner
 
Architectural design
Architectural design
SHREEHARI WADAWADAGI
 
Towards Software Sustainability Guides for Industrial Software Systems
Towards Software Sustainability Guides for Industrial Software Systems
Heiko Koziolek
 
Large-Scale Empirical Studies of Mobile Apps
Large-Scale Empirical Studies of Mobile Apps
SAIL_QU
 
Medios publicitarios
Medios publicitarios
Giarline
 
Iniciativa del Senador Luis Humberto FernĂĄndez sobre coaliciones entre indepe...
Iniciativa del Senador Luis Humberto FernĂĄndez sobre coaliciones entre indepe...
Senadores PRD
 

More Related Content

What's hot (19)

Software engineering- system testing
Software engineering- system testing
Tejas Mhaske
 
Testing Technique
Testing Technique
Ajeng Savitri
 
Documented Requirements are not Useless After All!
Documented Requirements are not Useless After All!
Lionel Briand
 
Testing Plan
Testing Plan
Ajeng Savitri
 
Testing throughout the software life cycle
Testing throughout the software life cycle
Alfa Rizki Harahap
 
Automated Change Impact Analysis between SysML Models of Requirements and Design
Automated Change Impact Analysis between SysML Models of Requirements and Design
Lionel Briand
 
Defect Prediction Over Software Life Cycle in Automotive Domain
Defect Prediction Over Software Life Cycle in Automotive Domain
RAKESH RANA
 
Survey on Software Defect Prediction
Survey on Software Defect Prediction
Sung Kim
 
naveed-kamran-software-architecture-agile
naveed-kamran-software-architecture-agile
Naveed Kamran
 
System development phases
System development phases
PrabinBista05
 
Chaper 1 sdlc
Chaper 1 sdlc
Azrul Aziz
 
Fundamentals of Software Engineering
Fundamentals of Software Engineering
Madhar Khan Pathan
 
Survey on Software Defect Prediction (PhD Qualifying Examination Presentation)
Survey on Software Defect Prediction (PhD Qualifying Examination Presentation)
lifove
 
Survey on Software Defect Prediction
Survey on Software Defect Prediction
lifove
 
Maintenance, Re-engineering &Reverse Engineering in Software Engineering
Maintenance, Re-engineering &Reverse Engineering in Software Engineering
Manish Kumar
 
Software engineering 21 integration testing
Software engineering 21 integration testing
Vaibhav Khanna
 
Neotys PAC - Bruno Da Silva
Neotys PAC - Bruno Da Silva
Neotys_Partner
 
Architectural design
Architectural design
SHREEHARI WADAWADAGI
 
Towards Software Sustainability Guides for Industrial Software Systems
Towards Software Sustainability Guides for Industrial Software Systems
Heiko Koziolek
 
Software engineering- system testing
Software engineering- system testing
Tejas Mhaske
 
Testing Technique
Testing Technique
Ajeng Savitri
 
Documented Requirements are not Useless After All!
Documented Requirements are not Useless After All!
Lionel Briand
 
Testing throughout the software life cycle
Testing throughout the software life cycle
Alfa Rizki Harahap
 
Automated Change Impact Analysis between SysML Models of Requirements and Design
Automated Change Impact Analysis between SysML Models of Requirements and Design
Lionel Briand
 
Defect Prediction Over Software Life Cycle in Automotive Domain
Defect Prediction Over Software Life Cycle in Automotive Domain
RAKESH RANA
 
Survey on Software Defect Prediction
Survey on Software Defect Prediction
Sung Kim
 
naveed-kamran-software-architecture-agile
naveed-kamran-software-architecture-agile
Naveed Kamran
 
System development phases
System development phases
PrabinBista05
 
Chaper 1 sdlc
Chaper 1 sdlc
Azrul Aziz
 
Fundamentals of Software Engineering
Fundamentals of Software Engineering
Madhar Khan Pathan
 
Survey on Software Defect Prediction (PhD Qualifying Examination Presentation)
Survey on Software Defect Prediction (PhD Qualifying Examination Presentation)
lifove
 
Survey on Software Defect Prediction
Survey on Software Defect Prediction
lifove
 
Maintenance, Re-engineering &Reverse Engineering in Software Engineering
Maintenance, Re-engineering &Reverse Engineering in Software Engineering
Manish Kumar
 
Software engineering 21 integration testing
Software engineering 21 integration testing
Vaibhav Khanna
 
Neotys PAC - Bruno Da Silva
Neotys PAC - Bruno Da Silva
Neotys_Partner
 
Towards Software Sustainability Guides for Industrial Software Systems
Towards Software Sustainability Guides for Industrial Software Systems
Heiko Koziolek
 

Viewers also liked (13)

Large-Scale Empirical Studies of Mobile Apps
Large-Scale Empirical Studies of Mobile Apps
SAIL_QU
 
Medios publicitarios
Medios publicitarios
Giarline
 
Iniciativa del Senador Luis Humberto FernĂĄndez sobre coaliciones entre indepe...
Iniciativa del Senador Luis Humberto FernĂĄndez sobre coaliciones entre indepe...
Senadores PRD
 
Proyecto de sociologia
Proyecto de sociologia
ruth Apellidos
 
Case History Carlo Cairoli
Case History Carlo Cairoli
Carlo Cairoli
 
C de marchi_tirocinio
C de marchi_tirocinio
Cladema
 
비아그라 판매 =&lt;7cc.kr>=비아그라 정품 판매~비아그라판매±비아그라 정품판매∏비아그라 50mg판매,프릴리지 50mg판매,흥분제 ...
비아그라 판매 =&lt;7cc.kr>=비아그라 정품 판매~비아그라판매±비아그라 정품판매∏비아그라 50mg판매,프릴리지 50mg판매,흥분제 ...
成 金
 
Importance of Eye Contact in Hospitality
Importance of Eye Contact in Hospitality
Asian Institute of Hospitality & Tourism
 
Singapore startup ecosystem and entrepreneur toolbox - Aug 2015
Singapore startup ecosystem and entrepreneur toolbox - Aug 2015
Arnaud Bonzom
 
Powering of bangladesh- Vision 2021
Powering of bangladesh- Vision 2021
Mukhlasur Rahman
 
An Industrial Case Study on the Automated Detection of Performance Regression...
An Industrial Case Study on the Automated Detection of Performance Regression...
SAIL_QU
 
Sustainability Day Leeds 2017
Sustainability Day Leeds 2017
4 All of Us
 
Lean Canvas evolved - FTE Canvas
Lean Canvas evolved - FTE Canvas
Gorilla Ventures Oy
 
Large-Scale Empirical Studies of Mobile Apps
Large-Scale Empirical Studies of Mobile Apps
SAIL_QU
 
Medios publicitarios
Medios publicitarios
Giarline
 
Iniciativa del Senador Luis Humberto FernĂĄndez sobre coaliciones entre indepe...
Iniciativa del Senador Luis Humberto FernĂĄndez sobre coaliciones entre indepe...
Senadores PRD
 
Proyecto de sociologia
Proyecto de sociologia
ruth Apellidos
 
Case History Carlo Cairoli
Case History Carlo Cairoli
Carlo Cairoli
 
C de marchi_tirocinio
C de marchi_tirocinio
Cladema
 
비아그라 판매 =&lt;7cc.kr>=비아그라 정품 판매~비아그라판매±비아그라 정품판매∏비아그라 50mg판매,프릴리지 50mg판매,흥분제 ...
비아그라 판매 =&lt;7cc.kr>=비아그라 정품 판매~비아그라판매±비아그라 정품판매∏비아그라 50mg판매,프릴리지 50mg판매,흥분제 ...
成 金
 
Singapore startup ecosystem and entrepreneur toolbox - Aug 2015
Singapore startup ecosystem and entrepreneur toolbox - Aug 2015
Arnaud Bonzom
 
Powering of bangladesh- Vision 2021
Powering of bangladesh- Vision 2021
Mukhlasur Rahman
 
An Industrial Case Study on the Automated Detection of Performance Regression...
An Industrial Case Study on the Automated Detection of Performance Regression...
SAIL_QU
 
Sustainability Day Leeds 2017
Sustainability Day Leeds 2017
4 All of Us
 
Lean Canvas evolved - FTE Canvas
Lean Canvas evolved - FTE Canvas
Gorilla Ventures Oy
 
Ad

Similar to Modeling the Performance of Ultra-Large-Scale Systems Using Layered Simulations (20)

3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile
Neotys
 
Eric Proegler Early Performance Testing from CAST2014
Eric Proegler Early Performance Testing from CAST2014
Eric Proegler
 
performancetestinganoverview-110206071921-phpapp02.pdf
performancetestinganoverview-110206071921-phpapp02.pdf
MAshok10
 
Performance Testing Using VS 2010 - Part 1
Performance Testing Using VS 2010 - Part 1
Mohamed Tarek
 
Continuous Performance Testing: The New Standard
Continuous Performance Testing: The New Standard
TechWell
 
Performance testing and rpt
Performance testing and rpt
SivaprasanthRentala1975
 
Alexander Podelko - Context-Driven Performance Testing
Alexander Podelko - Context-Driven Performance Testing
Neotys_Partner
 
Workload design[1]
Workload design[1]
Benyam Ayalew
 
Primer on application_performance_testing_v0.2
Primer on application_performance_testing_v0.2
Trevor Warren
 
What is performance_engineering_v0.2
What is performance_engineering_v0.2
Trevor Warren
 
Performance testing : An Overview
Performance testing : An Overview
sharadkjain
 
Best Practices for Applications Performance Testing
Best Practices for Applications Performance Testing
Bhaskara Reddy Sannapureddy
 
Performance Testing in the Agile Lifecycle
Performance Testing in the Agile Lifecycle
Lee Barnes
 
PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo
Neotys
 
The art of system and solution testing
The art of system and solution testing
gaoliang641
 
Time-resource v&v for complex systems
Time-resource v&v for complex systems
Predictable Network Solutions Ltd.
 
PEnDAR webinar 1 with notes
PEnDAR webinar 1 with notes
Predictable Network Solutions Ltd.
 
Performance testing: respect the difference at swqd14
Performance testing: respect the difference at swqd14
Alexander Podelko
 
Neotys PAC - Adding Performance Verifications in Continuous Delivery
Neotys PAC - Adding Performance Verifications in Continuous Delivery
Federico Toledo
 
Context-Driven Performance Testing
Context-Driven Performance Testing
Alexander Podelko
 
3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile
Neotys
 
Eric Proegler Early Performance Testing from CAST2014
Eric Proegler Early Performance Testing from CAST2014
Eric Proegler
 
performancetestinganoverview-110206071921-phpapp02.pdf
performancetestinganoverview-110206071921-phpapp02.pdf
MAshok10
 
Performance Testing Using VS 2010 - Part 1
Performance Testing Using VS 2010 - Part 1
Mohamed Tarek
 
Continuous Performance Testing: The New Standard
Continuous Performance Testing: The New Standard
TechWell
 
Alexander Podelko - Context-Driven Performance Testing
Alexander Podelko - Context-Driven Performance Testing
Neotys_Partner
 
Workload design[1]
Workload design[1]
Benyam Ayalew
 
Primer on application_performance_testing_v0.2
Primer on application_performance_testing_v0.2
Trevor Warren
 
What is performance_engineering_v0.2
What is performance_engineering_v0.2
Trevor Warren
 
Performance testing : An Overview
Performance testing : An Overview
sharadkjain
 
Best Practices for Applications Performance Testing
Best Practices for Applications Performance Testing
Bhaskara Reddy Sannapureddy
 
Performance Testing in the Agile Lifecycle
Performance Testing in the Agile Lifecycle
Lee Barnes
 
PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo
Neotys
 
The art of system and solution testing
The art of system and solution testing
gaoliang641
 
Performance testing: respect the difference at swqd14
Performance testing: respect the difference at swqd14
Alexander Podelko
 
Neotys PAC - Adding Performance Verifications in Continuous Delivery
Neotys PAC - Adding Performance Verifications in Continuous Delivery
Federico Toledo
 
Context-Driven Performance Testing
Context-Driven Performance Testing
Alexander Podelko
 
Ad

More from SAIL_QU (20)

Studying the Integration Practices and the Evolution of Ad Libraries in the G...
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
SAIL_QU
 
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
SAIL_QU
 
Improving the testing efficiency of selenium-based load tests
Improving the testing efficiency of selenium-based load tests
SAIL_QU
 
Studying User-Developer Interactions Through the Distribution and Reviewing M...
Studying User-Developer Interactions Through the Distribution and Reviewing M...
SAIL_QU
 
Studying online distribution platforms for games through the mining of data f...
Studying online distribution platforms for games through the mining of data f...
SAIL_QU
 
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
SAIL_QU
 
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
SAIL_QU
 
Mining Development Knowledge to Understand and Support Software Logging Pract...
Mining Development Knowledge to Understand and Support Software Logging Pract...
SAIL_QU
 
Which Log Level Should Developers Choose For a New Logging Statement?
Which Log Level Should Developers Choose For a New Logging Statement?
SAIL_QU
 
Towards Just-in-Time Suggestions for Log Changes
Towards Just-in-Time Suggestions for Log Changes
SAIL_QU
 
The Impact of Task Granularity on Co-evolution Analyses
The Impact of Task Granularity on Co-evolution Analyses
SAIL_QU
 
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
SAIL_QU
 
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
SAIL_QU
 
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
SAIL_QU
 
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
SAIL_QU
 
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
SAIL_QU
 
What Do Programmers Know about Software Energy Consumption?
What Do Programmers Know about Software Energy Consumption?
SAIL_QU
 
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
SAIL_QU
 
Revisiting the Experimental Design Choices for Approaches for the Automated R...
Revisiting the Experimental Design Choices for Approaches for the Automated R...
SAIL_QU
 
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
SAIL_QU
 
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
Studying the Integration Practices and the Evolution of Ad Libraries in the G...
SAIL_QU
 
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
SAIL_QU
 
Improving the testing efficiency of selenium-based load tests
Improving the testing efficiency of selenium-based load tests
SAIL_QU
 
Studying User-Developer Interactions Through the Distribution and Reviewing M...
Studying User-Developer Interactions Through the Distribution and Reviewing M...
SAIL_QU
 
Studying online distribution platforms for games through the mining of data f...
Studying online distribution platforms for games through the mining of data f...
SAIL_QU
 
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
Understanding the Factors for Fast Answers in Technical Q&A Websites: An Empi...
SAIL_QU
 
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
Investigating the Challenges in Selenium Usage and Improving the Testing Effi...
SAIL_QU
 
Mining Development Knowledge to Understand and Support Software Logging Pract...
Mining Development Knowledge to Understand and Support Software Logging Pract...
SAIL_QU
 
Which Log Level Should Developers Choose For a New Logging Statement?
Which Log Level Should Developers Choose For a New Logging Statement?
SAIL_QU
 
Towards Just-in-Time Suggestions for Log Changes
Towards Just-in-Time Suggestions for Log Changes
SAIL_QU
 
The Impact of Task Granularity on Co-evolution Analyses
The Impact of Task Granularity on Co-evolution Analyses
SAIL_QU
 
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
A Framework for Evaluating the Results of the SZZ Approach for Identifying Bu...
SAIL_QU
 
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
How are Discussions Associated with Bug Reworking? An Empirical Study on Open...
SAIL_QU
 
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
A Study of the Relation of Mobile Device Attributes with the User-Perceived Q...
SAIL_QU
 
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
A Large-Scale Study of the Impact of Feature Selection Techniques on Defect C...
SAIL_QU
 
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
Studying the Dialogue Between Users and Developers of Free Apps in the Google...
SAIL_QU
 
What Do Programmers Know about Software Energy Consumption?
What Do Programmers Know about Software Energy Consumption?
SAIL_QU
 
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
Threshold for Size and Complexity Metrics: A Case Study from the Perspective ...
SAIL_QU
 
Revisiting the Experimental Design Choices for Approaches for the Automated R...
Revisiting the Experimental Design Choices for Approaches for the Automated R...
SAIL_QU
 
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
SAIL_QU
 

Recently uploaded (20)

Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Philip Schwarz
 
Azure AI Foundry: The AI app and agent factory
Azure AI Foundry: The AI app and agent factory
Maxim Salnikov
 
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
IFI Techsolutions
 
From Code to Commerce, a Backend Java Developer's Galactic Journey into Ecomm...
From Code to Commerce, a Backend Java Developer's Galactic Journey into Ecomm...
Jamie Coleman
 
Download Adobe Illustrator Crack free for Windows 2025?
Download Adobe Illustrator Crack free for Windows 2025?
grete1122g
 
A Guide to Telemedicine Software Development.pdf
A Guide to Telemedicine Software Development.pdf
Olivero Bozzelli
 
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
arabelatso
 
Sysinfo OST to PST Converter Infographic
Sysinfo OST to PST Converter Infographic
SysInfo Tools
 
Advance Doctor Appointment Booking App With Online Payment
Advance Doctor Appointment Booking App With Online Payment
AxisTechnolabs
 
Building Geospatial Data Warehouse for GIS by GIS with FME
Building Geospatial Data Warehouse for GIS by GIS with FME
Safe Software
 
OpenChain Webinar - AboutCode - Practical Compliance in One Stack – Licensing...
OpenChain Webinar - AboutCode - Practical Compliance in One Stack – Licensing...
Shane Coughlan
 
From Data Preparation to Inference: How Alluxio Speeds Up AI
From Data Preparation to Inference: How Alluxio Speeds Up AI
Alluxio, Inc.
 
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
pcprocore
 
Simplify Task, Team, and Project Management with Orangescrum Work
Simplify Task, Team, and Project Management with Orangescrum Work
Orangescrum
 
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
WSO2
 
Best Practice for LLM Serving in the Cloud
Best Practice for LLM Serving in the Cloud
Alluxio, Inc.
 
Complete WordPress Programming Guidance Book
Complete WordPress Programming Guidance Book
Shabista Imam
 
Streamlining CI/CD with FME Flow: A Practical Guide
Streamlining CI/CD with FME Flow: A Practical Guide
Safe Software
 
declaration of Variables and constants.pptx
declaration of Variables and constants.pptx
meemee7378
 
Key Challenges in Troubleshooting Customer On-Premise Applications
Key Challenges in Troubleshooting Customer On-Premise Applications
Tier1 app
 
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Philip Schwarz
 
Azure AI Foundry: The AI app and agent factory
Azure AI Foundry: The AI app and agent factory
Maxim Salnikov
 
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
IFI Techsolutions
 
From Code to Commerce, a Backend Java Developer's Galactic Journey into Ecomm...
From Code to Commerce, a Backend Java Developer's Galactic Journey into Ecomm...
Jamie Coleman
 
Download Adobe Illustrator Crack free for Windows 2025?
Download Adobe Illustrator Crack free for Windows 2025?
grete1122g
 
A Guide to Telemedicine Software Development.pdf
A Guide to Telemedicine Software Development.pdf
Olivero Bozzelli
 
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
arabelatso
 
Sysinfo OST to PST Converter Infographic
Sysinfo OST to PST Converter Infographic
SysInfo Tools
 
Advance Doctor Appointment Booking App With Online Payment
Advance Doctor Appointment Booking App With Online Payment
AxisTechnolabs
 
Building Geospatial Data Warehouse for GIS by GIS with FME
Building Geospatial Data Warehouse for GIS by GIS with FME
Safe Software
 
OpenChain Webinar - AboutCode - Practical Compliance in One Stack – Licensing...
OpenChain Webinar - AboutCode - Practical Compliance in One Stack – Licensing...
Shane Coughlan
 
From Data Preparation to Inference: How Alluxio Speeds Up AI
From Data Preparation to Inference: How Alluxio Speeds Up AI
Alluxio, Inc.
 
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
pcprocore
 
Simplify Task, Team, and Project Management with Orangescrum Work
Simplify Task, Team, and Project Management with Orangescrum Work
Orangescrum
 
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
Modern Platform Engineering with Choreo - The AI-Native Internal Developer Pl...
WSO2
 
Best Practice for LLM Serving in the Cloud
Best Practice for LLM Serving in the Cloud
Alluxio, Inc.
 
Complete WordPress Programming Guidance Book
Complete WordPress Programming Guidance Book
Shabista Imam
 
Streamlining CI/CD with FME Flow: A Practical Guide
Streamlining CI/CD with FME Flow: A Practical Guide
Safe Software
 
declaration of Variables and constants.pptx
declaration of Variables and constants.pptx
meemee7378
 
Key Challenges in Troubleshooting Customer On-Premise Applications
Key Challenges in Troubleshooting Customer On-Premise Applications
Tier1 app
 

Modeling the Performance of Ultra-Large-Scale Systems Using Layered Simulations

  • 1. Modeling the Performance of Ultra-Large-Scale Systems 1 Using Layered Simulations
  • 2. [Slide courtesy of Randy Shoup, eBay Distinguished Architect] 2 ULS are Everywhere, and they’re HUGE!
  • 3. 1. Performance Testing should happen … 4 development process here? here!
  • 4. 2. One Model to Rule Them All? 1. How do response time differ between these two mechanism? 5
  • 5. 3. Different Stakeholders, Different Interests End User Developer System Engineer System Integrator Sales Rep 6
  • 6. An Analogy: The “4+1” View Model Scenario Logical View Process View Development View Physical View 8
  • 7. The “3+1” View Model for Performance Modeling Scenario World View Physical View Component View 9
  • 8. Component Layer A Layered Simulation Model! 10 World View Layer Physical Layer
  • 9. Case Study 1 Bottleneck Detection in RSS Cloud • RSS blogging service providers (e.g., Wordpress) • Evaluation of two notification delivery mechanisms: – Periodic Pull – Real-time Push Q1. How does response time differ? Q2. How does resource usage (e.g., CPU, memory) look like? 11
  • 10. Component Layer The RSS Cloud Model 12 World View Layer Physical Layer
  • 11. CPU is the Bottleneck! 13
  • 12. Case Study 2 Evaluating a Distributed Monitor 1. How do response time differ between these two mechanism? 14
  • 15. Keeping the Model Up-to-Date • (What-if) Experimenting with different configuration/deployment scenarios • (Digging deeper) Adding new “layers”, as system is understood better • (Design evolution) Updating the simulation models as the system evolves 18
  • 16. GH'@&+/) +A 12(&'I &-829'-3) *#%'3144&2'J ' 6! 1) +) 789: ) 02! 9(8; ) <<! 3&+&K' 3&+&F' LH'7$M&+&2,'",1N&3) #%&+-5'7$M&+&2,'O2,&+&-,-' ?2%'= -&+' 7&B&#) 4&+' ". -,&A '' ?29$2&&+' ". -,&A '' O2,&9+1,) +' "1#&-'' 0&4' =! Component Layer' : 'Z1. &+&%'"$A *#18) 2'P ) %&#F' "I ! World View Layer' Physical Layer' 7$-,+$S*,&%'B-H'&2,+1#$d&%! "F! eHG<d' eHL<d' eHQ<d' TABLE IX SIMULATION RESULTS FOR THE ORIGINAL DESIGN OF THE PERFORMANCE MONITOR. DataCollection Response time(s) Cost per Central Monitor Central Monitor Central Monitor Frequency (Hz) transmission ($) Thread Util. (%) CPU Util. (%) RAM Util. (%) 0.1 0.2 2.2 31.0 42.6 37.2 0.2 0.3 2.7 43.7 68.2 47.6 0.3 0.4 3.1 60.2 86.6 59.2 the CPU of the central monitor is close to running at its full capacity, which will likely result in system instability. More- over, whiletheoriginal designhaslower cost per transmission, be validated as information becomes available throug development. Due to the lack of access to production we could only estimate a list of resources and proce eHG<d' eHL<d' eHQ<d' /1-,&+! U(3&14&+X! #) D&+'+&-) *+(&'*-19&! QUESTIONS!
  • 17. 20
  • 19. FOUR CHALLENGES The Current Practice of Performance Verification 22
  • 20. 1. Too Late in the Development Lifecycle • Design changes are not evaluated until after code is written – Happens at the last stage of a delayed schedule 23 Requirements Design Implementation Verification Maintenance
  • 21. 2. Lots of Data • Industrial case studies have 2000> counters • Time consuming to analyze • Hard to compare more than 2 tests at once 24
  • 22. 3. No Documented Behavior 25 • Analysts have different perceptions of performance regressions • Analysis may be influenced by – Analyst’s knowledge – Deadline
  • 23. 4. Heterogeneous Environments • Multiple labs to parallelize test executions – Hardware and software may differ – Tests from one lab may not be used to analyze tests from another lab 26
  • 25. AT THE DESIGN LEVEL Performance Verification 28
  • 26. Evaluate Design Changes through Performance Modeling • Analytical models are often not suitable for all stakeholders – Abstract mathematical and statistical concepts • Simulation models can be implemented with support of existing framework – Visualization – No systematic approach to construct models that can be used by different stakeholders 29
  • 27. Layered Simulation Model 30 Physical layerComponent layer World view layer Can the current infrastructure support the projected growth of users? Investigate threading model Hardware resource utilization
  • 28. Case Studies • We conducted two case studies – RSS Cloud • Show the process of constructed the model • Derive the bottleneck of the application – Performance monitor for ULS systems • Evaluate whether or not an organization should re- architect the software • Our model can be used to extract important information and aid in decision making 31

Editor's Notes

  • #3: Slides from eBay to give another taste of scale
  • #7: Spend a bit of time about performance concerns. E.g. End user: overall system performance for various configurations Developer: Organization and performance of system modules System Engineer: Hardware resource util, hardware capacity planning System Integrator: Performance after integration of different components Sales Representative: Reliability, costs
  • #8: Analytical Modeling: use math equations and statistical concepts to model the performance of a software system. This requires substantial level of expertise. Simulation models (Discrete-event simulation) can created to answer specific performance questions from different stakeholders.
  • #10: Explain a bit on what the models are and who those stake holders are
  • #11: Discuss the contents in Table III
  • #13: Discuss the contents in Table III