SlideShare a Scribd company logo
The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare
In the next half hour… Partners Healthcare System overview Caché platform architecture & metrics The need to migrate Phased migration approach Benchmark testing and results Discoveries and production enhancements
Partners Healthcare System Founded in 1994  Brigham & Women’s Hospital  Massachusetts General Hospital Now includes: Community physician network (1200 + 3500 MD’s) PCHi 3 community hospitals 2 rehab hospitals 3 specialty institutions  Enterprise-wide Information Systems 1100 employees Annual budget FY05 approximately $160 million
Anchor Hospitals & Airport BWH MGH Logan Airport 10 km 6 km
Acute Care Hospitals MGH BWH Newton- Wellesley Community Physician Practices
Partners Domain Devices Internet 12,000 Printers 32,000 Desktops Firewall ~30,000 other devices 1,450 Servers Closely Managed Assumed Managed
Windows Production Architecture 3.5 TB
Enterprise Integration Over 30% are to and from Caché database Change from prior year Daily  Average Est. Annual Transactions # of Interfaces 196 170 192 167 37% 4,659,035 1,330,962,017 2007 40% 3,399,211 1,240,712,044 2006 45% 2,431,917 887,649,802 2005 1,673,515 610,833,080 2004
Integration Components
Gigabytes in Use
Annual Database Growth Rate
Database Utilization Average Database References per day in Billions
The Need to Migrate - Availability Monthly Downtime Current  State Business  need
Additional Business Requirements Increase availability and reliability Decrease database risk from 5 single points of failure More robust hardware and OS Many less servers and OS instances to manage Clustering and automated failover  Reduce monthly maintenance needs, updates once or twice per year -------------------------------------------------------- Improve Performance  64 bit OS, more memory for cache Caché 5.0.20 to Caché 2008.1, significantly improved  ECP performance Increase Scalability 91 Terabytes available on EMC SAN DMX3 On-demand addition of processor cores
Caché Migration Decision Making Process Only considered first tier vendors and support  (IBM, HP) HP assumed much more risk with Professional  Services Existing HP business yields more leverage & visibility with regional office More headroom in HP configuration Price was not a distinguishing factor
Phased migration approach Proof of Concept (benchmark testing) Completed 10/15/07 Phase 1 – Database tier 4 of 5 servers migrated, anticipated completion 4/14/08 Phase 2 – Application tier Big Bang migration 12/14/08 Phase 3 – Disaster Recovery January 2009
UNIX Benchmark Environment
Database Benchmark Load Testing Results Goals  Simulate current Production user counts & transaction loads Verify support for load increases up to 300% Benchmark Environment Isolated LAN, new DMX3 SAN 20 new Windows blade servers (10 app servers, 10 script ‘players’) Scripts for 8 apps (represent heaviest use, Web/Telnet/VB apps) 2 batch jobs (screensaver simulation, NullGen LMR functions) Conclusions Able to simulate production load, 1.5x and 3x load 2 HP rx8640 can handle growth projections 0.66 0.15 0.32 LMR avg Caché app time (in sec.) 40,000 40,000 11,806 LMR transactions (5 min. period) 135,000 30,000 35,000 Database Global Refs / sec. Benchmark full script load Benchmark “paced” script load Production peak (8/21, 11:20 am) Metric
Design and Configuration Considerations Database configuration simulation testing 1 to 5 Caché database instances were assessed 1 vs. 5 ECP channels per Caché instance were assessed Number of active cores were accessed (4 active, 2  reserved) Results and unexpected discoveries Identify 5 Caché database instance as optimal design configuration Journal synch bottleneck the biggest issue  High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance) Maintain same data distribution across 5 DB instances Determine 1 ECP channel per instance optimal Additional channels did not improve throughput, still have only 1 Journal Deamon
Benchmark Discoveries led to Production Improvements References to  Undefined  globals using $Data and $Get  These commands require network round trip Use of $increment  Each call to $I requires network round trip Excessive use of Cache locks  Forces more than 1 round trip Use of large strings  Strings that require more than 3900–4000 bytes to represent the string value are big strings and never cached on the ECP client.   Lesson Learned -  Each trip to the database server results in overhead caused by a Journal Synch.  Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions .
75% reduction in long running transaction
Phased Migration Approach
Monthly Average Caché Web Transaction Time
Application Models Old   New Browser  client Web server Cache Cache VB client .Net server Cache Cache .Net client Browser  client Web server Cache Web Services Browser  client .Net client Scalability/Connection pooling, robustness/error handling, Vism Managed Obj. Vism.ocx Managed Obj. Cache Web Services WebLink
The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare
Ad

More Related Content

What's hot (20)

2009 - Microsoft IIS Vs. Apache - Who Serves More - A Study
2009 - Microsoft IIS Vs. Apache - Who Serves More - A Study2009 - Microsoft IIS Vs. Apache - Who Serves More - A Study
2009 - Microsoft IIS Vs. Apache - Who Serves More - A Study
Vijay Prasad Gupta
 
Optimiszing proxy
Optimiszing proxyOptimiszing proxy
Optimiszing proxy
Proxies Rent
 
Coherence sig-nfr-web-tier-scaling-using-coherence-web
Coherence sig-nfr-web-tier-scaling-using-coherence-webCoherence sig-nfr-web-tier-scaling-using-coherence-web
Coherence sig-nfr-web-tier-scaling-using-coherence-web
C2B2 Consulting
 
Web Server Technologies I: HTTP & Getting Started
Web Server Technologies I: HTTP & Getting StartedWeb Server Technologies I: HTTP & Getting Started
Web Server Technologies I: HTTP & Getting Started
Port80 Software
 
Web ,app and db server presentation
Web ,app and db server presentationWeb ,app and db server presentation
Web ,app and db server presentation
Parth Godhani
 
Web server
Web serverWeb server
Web server
Nirav Daraniya
 
Optimizing proxy
Optimizing proxyOptimizing proxy
Optimizing proxy
Proxies Rent
 
Web server administration
Web server administrationWeb server administration
Web server administration
sawsan slii
 
Datasheet weblogic midvisionextensionforibmraf
Datasheet weblogic midvisionextensionforibmrafDatasheet weblogic midvisionextensionforibmraf
Datasheet weblogic midvisionextensionforibmraf
MidVision
 
Web Server Technologies II: Web Applications & Server Maintenance
Web Server Technologies II: Web Applications & Server MaintenanceWeb Server Technologies II: Web Applications & Server Maintenance
Web Server Technologies II: Web Applications & Server Maintenance
Port80 Software
 
Performance and Scalability Tuning
Performance and Scalability TuningPerformance and Scalability Tuning
Performance and Scalability Tuning
Andres March
 
Web Server Hardware and Software
Web Server Hardware and SoftwareWeb Server Hardware and Software
Web Server Hardware and Software
webhostingguy
 
Bluedog white paper - scaling for high availability, high utilization
Bluedog white paper - scaling for high availability, high utilizationBluedog white paper - scaling for high availability, high utilization
Bluedog white paper - scaling for high availability, high utilization
tom termini
 
HTML5 Server Sent Events/JSF JAX 2011 Conference
HTML5 Server Sent Events/JSF  JAX 2011 ConferenceHTML5 Server Sent Events/JSF  JAX 2011 Conference
HTML5 Server Sent Events/JSF JAX 2011 Conference
Roger Kitain
 
Weblogic-clustering-failover-and-load-balancing-training
Weblogic-clustering-failover-and-load-balancing-trainingWeblogic-clustering-failover-and-load-balancing-training
Weblogic-clustering-failover-and-load-balancing-training
Unmesh Baile
 
Oracle WebLogic Server Basic Concepts
Oracle WebLogic Server Basic ConceptsOracle WebLogic Server Basic Concepts
Oracle WebLogic Server Basic Concepts
James Bayer
 
Web servers – features, installation and configuration
Web servers – features, installation and configurationWeb servers – features, installation and configuration
Web servers – features, installation and configuration
webhostingguy
 
LinkedIn - A highly scalable Architecture on Java!
LinkedIn - A highly scalable Architecture on Java!LinkedIn - A highly scalable Architecture on Java!
LinkedIn - A highly scalable Architecture on Java!
manivannan57
 
Building a Scalable Architecture for web apps
Building a Scalable Architecture for web appsBuilding a Scalable Architecture for web apps
Building a Scalable Architecture for web apps
Directi Group
 
ASP.NET Web API and HTTP Fundamentals
ASP.NET Web API and HTTP FundamentalsASP.NET Web API and HTTP Fundamentals
ASP.NET Web API and HTTP Fundamentals
Ido Flatow
 
2009 - Microsoft IIS Vs. Apache - Who Serves More - A Study
2009 - Microsoft IIS Vs. Apache - Who Serves More - A Study2009 - Microsoft IIS Vs. Apache - Who Serves More - A Study
2009 - Microsoft IIS Vs. Apache - Who Serves More - A Study
Vijay Prasad Gupta
 
Coherence sig-nfr-web-tier-scaling-using-coherence-web
Coherence sig-nfr-web-tier-scaling-using-coherence-webCoherence sig-nfr-web-tier-scaling-using-coherence-web
Coherence sig-nfr-web-tier-scaling-using-coherence-web
C2B2 Consulting
 
Web Server Technologies I: HTTP & Getting Started
Web Server Technologies I: HTTP & Getting StartedWeb Server Technologies I: HTTP & Getting Started
Web Server Technologies I: HTTP & Getting Started
Port80 Software
 
Web ,app and db server presentation
Web ,app and db server presentationWeb ,app and db server presentation
Web ,app and db server presentation
Parth Godhani
 
Web server administration
Web server administrationWeb server administration
Web server administration
sawsan slii
 
Datasheet weblogic midvisionextensionforibmraf
Datasheet weblogic midvisionextensionforibmrafDatasheet weblogic midvisionextensionforibmraf
Datasheet weblogic midvisionextensionforibmraf
MidVision
 
Web Server Technologies II: Web Applications & Server Maintenance
Web Server Technologies II: Web Applications & Server MaintenanceWeb Server Technologies II: Web Applications & Server Maintenance
Web Server Technologies II: Web Applications & Server Maintenance
Port80 Software
 
Performance and Scalability Tuning
Performance and Scalability TuningPerformance and Scalability Tuning
Performance and Scalability Tuning
Andres March
 
Web Server Hardware and Software
Web Server Hardware and SoftwareWeb Server Hardware and Software
Web Server Hardware and Software
webhostingguy
 
Bluedog white paper - scaling for high availability, high utilization
Bluedog white paper - scaling for high availability, high utilizationBluedog white paper - scaling for high availability, high utilization
Bluedog white paper - scaling for high availability, high utilization
tom termini
 
HTML5 Server Sent Events/JSF JAX 2011 Conference
HTML5 Server Sent Events/JSF  JAX 2011 ConferenceHTML5 Server Sent Events/JSF  JAX 2011 Conference
HTML5 Server Sent Events/JSF JAX 2011 Conference
Roger Kitain
 
Weblogic-clustering-failover-and-load-balancing-training
Weblogic-clustering-failover-and-load-balancing-trainingWeblogic-clustering-failover-and-load-balancing-training
Weblogic-clustering-failover-and-load-balancing-training
Unmesh Baile
 
Oracle WebLogic Server Basic Concepts
Oracle WebLogic Server Basic ConceptsOracle WebLogic Server Basic Concepts
Oracle WebLogic Server Basic Concepts
James Bayer
 
Web servers – features, installation and configuration
Web servers – features, installation and configurationWeb servers – features, installation and configuration
Web servers – features, installation and configuration
webhostingguy
 
LinkedIn - A highly scalable Architecture on Java!
LinkedIn - A highly scalable Architecture on Java!LinkedIn - A highly scalable Architecture on Java!
LinkedIn - A highly scalable Architecture on Java!
manivannan57
 
Building a Scalable Architecture for web apps
Building a Scalable Architecture for web appsBuilding a Scalable Architecture for web apps
Building a Scalable Architecture for web apps
Directi Group
 
ASP.NET Web API and HTTP Fundamentals
ASP.NET Web API and HTTP FundamentalsASP.NET Web API and HTTP Fundamentals
ASP.NET Web API and HTTP Fundamentals
Ido Flatow
 

Viewers also liked (11)

Certified Healthcare Network
Certified Healthcare NetworkCertified Healthcare Network
Certified Healthcare Network
phelixx372
 
Example of a social network chart
Example of a social network chartExample of a social network chart
Example of a social network chart
Cindy Tart
 
Create Your Own Social Network with Ning
Create Your Own Social Network with NingCreate Your Own Social Network with Ning
Create Your Own Social Network with Ning
Bethany Smith
 
NHS Confederation Future Healthcare Network
NHS Confederation Future Healthcare NetworkNHS Confederation Future Healthcare Network
NHS Confederation Future Healthcare Network
Lean Enterprise Academy
 
Access to Healthcare Network Member Manual
Access to Healthcare Network Member ManualAccess to Healthcare Network Member Manual
Access to Healthcare Network Member Manual
AccessToHealthcare
 
Social network analysis course 2010 - 2011
Social network analysis course 2010 - 2011Social network analysis course 2010 - 2011
Social network analysis course 2010 - 2011
guillaume ereteo
 
I Social Network
I Social NetworkI Social Network
I Social Network
Riva Giuseppe
 
The Basics of Social Network Analysis
The Basics of Social Network AnalysisThe Basics of Social Network Analysis
The Basics of Social Network Analysis
Rory Sie
 
Social network analysis
Social network analysisSocial network analysis
Social network analysis
Caleb Jones
 
Communication Networks
Communication NetworksCommunication Networks
Communication Networks
Alex Zagoumenov
 
Social Network Analysis
Social Network AnalysisSocial Network Analysis
Social Network Analysis
Giorgos Cheliotis
 
Certified Healthcare Network
Certified Healthcare NetworkCertified Healthcare Network
Certified Healthcare Network
phelixx372
 
Example of a social network chart
Example of a social network chartExample of a social network chart
Example of a social network chart
Cindy Tart
 
Create Your Own Social Network with Ning
Create Your Own Social Network with NingCreate Your Own Social Network with Ning
Create Your Own Social Network with Ning
Bethany Smith
 
NHS Confederation Future Healthcare Network
NHS Confederation Future Healthcare NetworkNHS Confederation Future Healthcare Network
NHS Confederation Future Healthcare Network
Lean Enterprise Academy
 
Access to Healthcare Network Member Manual
Access to Healthcare Network Member ManualAccess to Healthcare Network Member Manual
Access to Healthcare Network Member Manual
AccessToHealthcare
 
Social network analysis course 2010 - 2011
Social network analysis course 2010 - 2011Social network analysis course 2010 - 2011
Social network analysis course 2010 - 2011
guillaume ereteo
 
The Basics of Social Network Analysis
The Basics of Social Network AnalysisThe Basics of Social Network Analysis
The Basics of Social Network Analysis
Rory Sie
 
Social network analysis
Social network analysisSocial network analysis
Social network analysis
Caleb Jones
 
Ad

Similar to The experiences of migrating a large scale, high performance healthcare network (20)

The experiences of migrating a large scale, high performance healthcare network
The experiences of migrating a large scale, high performance healthcare networkThe experiences of migrating a large scale, high performance healthcare network
The experiences of migrating a large scale, high performance healthcare network
george.james
 
Oracle Coherence: in-memory datagrid
Oracle Coherence: in-memory datagridOracle Coherence: in-memory datagrid
Oracle Coherence: in-memory datagrid
Emiliano Pecis
 
T3 Consortium's Performance Center of Excellence
T3 Consortium's Performance Center of ExcellenceT3 Consortium's Performance Center of Excellence
T3 Consortium's Performance Center of Excellence
veehikle
 
Case Study: How Cisco Gained Visibility into Network Utilization and Proacti...
Case Study:  How Cisco Gained Visibility into Network Utilization and Proacti...Case Study:  How Cisco Gained Visibility into Network Utilization and Proacti...
Case Study: How Cisco Gained Visibility into Network Utilization and Proacti...
CA Technologies
 
Hpe service virtualization 3.8 what's new chicago adm
Hpe service virtualization 3.8 what's new chicago admHpe service virtualization 3.8 what's new chicago adm
Hpe service virtualization 3.8 what's new chicago adm
Jeffrey Nunn
 
Introduction To Cloud Computing
Introduction To Cloud ComputingIntroduction To Cloud Computing
Introduction To Cloud Computing
Rinat Shagisultanov
 
Performance testing virtualized systems v5
Performance testing virtualized systems v5Performance testing virtualized systems v5
Performance testing virtualized systems v5
Mentora
 
Grid Economics for the Next Generation Data Center
Grid  Economics for the Next Generation Data CenterGrid  Economics for the Next Generation Data Center
Grid Economics for the Next Generation Data Center
George Demarest
 
Webinar: Deploying the Combined Virtual and Physical Infrastructure
Webinar: Deploying the Combined Virtual and Physical InfrastructureWebinar: Deploying the Combined Virtual and Physical Infrastructure
Webinar: Deploying the Combined Virtual and Physical Infrastructure
Pepperweed Consulting
 
Asynchronous Mobile Web Services:
Asynchronous Mobile Web Services: Asynchronous Mobile Web Services:
Asynchronous Mobile Web Services:
Dr. Fahad Aijaz
 
3 Hyper V
3 Hyper V3 Hyper V
3 Hyper V
johnbakerMS
 
ScalabilityAvailability
ScalabilityAvailabilityScalabilityAvailability
ScalabilityAvailability
webuploader
 
Hp Connect 10 06 08 V5
Hp Connect 10 06 08 V5Hp Connect 10 06 08 V5
Hp Connect 10 06 08 V5
guestea711d0
 
Java Abs Dynamic Server Replication
Java Abs   Dynamic Server ReplicationJava Abs   Dynamic Server Replication
Java Abs Dynamic Server Replication
ncct
 
Virtualization Business Case
Virtualization Business CaseVirtualization Business Case
Virtualization Business Case
TopLine Strategies
 
Star98
Star98Star98
Star98
Steve Oubre
 
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Prolifics
 
Real Time Network Monitoring System
Real  Time  Network  Monitoring  SystemReal  Time  Network  Monitoring  System
Real Time Network Monitoring System
Girish Naik
 
Aceleracion de aplicacione 2
Aceleracion de aplicacione 2Aceleracion de aplicacione 2
Aceleracion de aplicacione 2
jfth
 
Sunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Sunx4450 Intel7460 GigaSpaces XAP Platform BenchmarkSunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Sunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Shay Hassidim
 
The experiences of migrating a large scale, high performance healthcare network
The experiences of migrating a large scale, high performance healthcare networkThe experiences of migrating a large scale, high performance healthcare network
The experiences of migrating a large scale, high performance healthcare network
george.james
 
Oracle Coherence: in-memory datagrid
Oracle Coherence: in-memory datagridOracle Coherence: in-memory datagrid
Oracle Coherence: in-memory datagrid
Emiliano Pecis
 
T3 Consortium's Performance Center of Excellence
T3 Consortium's Performance Center of ExcellenceT3 Consortium's Performance Center of Excellence
T3 Consortium's Performance Center of Excellence
veehikle
 
Case Study: How Cisco Gained Visibility into Network Utilization and Proacti...
Case Study:  How Cisco Gained Visibility into Network Utilization and Proacti...Case Study:  How Cisco Gained Visibility into Network Utilization and Proacti...
Case Study: How Cisco Gained Visibility into Network Utilization and Proacti...
CA Technologies
 
Hpe service virtualization 3.8 what's new chicago adm
Hpe service virtualization 3.8 what's new chicago admHpe service virtualization 3.8 what's new chicago adm
Hpe service virtualization 3.8 what's new chicago adm
Jeffrey Nunn
 
Performance testing virtualized systems v5
Performance testing virtualized systems v5Performance testing virtualized systems v5
Performance testing virtualized systems v5
Mentora
 
Grid Economics for the Next Generation Data Center
Grid  Economics for the Next Generation Data CenterGrid  Economics for the Next Generation Data Center
Grid Economics for the Next Generation Data Center
George Demarest
 
Webinar: Deploying the Combined Virtual and Physical Infrastructure
Webinar: Deploying the Combined Virtual and Physical InfrastructureWebinar: Deploying the Combined Virtual and Physical Infrastructure
Webinar: Deploying the Combined Virtual and Physical Infrastructure
Pepperweed Consulting
 
Asynchronous Mobile Web Services:
Asynchronous Mobile Web Services: Asynchronous Mobile Web Services:
Asynchronous Mobile Web Services:
Dr. Fahad Aijaz
 
ScalabilityAvailability
ScalabilityAvailabilityScalabilityAvailability
ScalabilityAvailability
webuploader
 
Hp Connect 10 06 08 V5
Hp Connect 10 06 08 V5Hp Connect 10 06 08 V5
Hp Connect 10 06 08 V5
guestea711d0
 
Java Abs Dynamic Server Replication
Java Abs   Dynamic Server ReplicationJava Abs   Dynamic Server Replication
Java Abs Dynamic Server Replication
ncct
 
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Prolifics
 
Real Time Network Monitoring System
Real  Time  Network  Monitoring  SystemReal  Time  Network  Monitoring  System
Real Time Network Monitoring System
Girish Naik
 
Aceleracion de aplicacione 2
Aceleracion de aplicacione 2Aceleracion de aplicacione 2
Aceleracion de aplicacione 2
jfth
 
Sunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Sunx4450 Intel7460 GigaSpaces XAP Platform BenchmarkSunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Sunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Shay Hassidim
 
Ad

More from george.james (20)

Fosdem 2010 GT.M and OpenStreetMap
Fosdem 2010 GT.M and OpenStreetMapFosdem 2010 GT.M and OpenStreetMap
Fosdem 2010 GT.M and OpenStreetMap
george.james
 
M/DB and M/DB:X
M/DB and M/DB:XM/DB and M/DB:X
M/DB and M/DB:X
george.james
 
Lost In The Clouds
Lost In The CloudsLost In The Clouds
Lost In The Clouds
george.james
 
On a cloudy day you can scale forever
On a cloudy day you can scale foreverOn a cloudy day you can scale forever
On a cloudy day you can scale forever
george.james
 
Bad Light Stops Play
Bad Light Stops PlayBad Light Stops Play
Bad Light Stops Play
george.james
 
Securing The Cloud
Securing The CloudSecuring The Cloud
Securing The Cloud
george.james
 
Out Of The Slipstream Proposal
Out Of The Slipstream ProposalOut Of The Slipstream Proposal
Out Of The Slipstream Proposal
george.james
 
Lightning In The Clouds
Lightning In The CloudsLightning In The Clouds
Lightning In The Clouds
george.james
 
Lost In The Clouds
Lost In The CloudsLost In The Clouds
Lost In The Clouds
george.james
 
Mumps the Internet scale database
Mumps the Internet scale databaseMumps the Internet scale database
Mumps the Internet scale database
george.james
 
Web Development Environments: Choose the best or go with the rest
Web Development Environments:  Choose the best or go with the restWeb Development Environments:  Choose the best or go with the rest
Web Development Environments: Choose the best or go with the rest
george.james
 
Google's BigTable
Google's BigTableGoogle's BigTable
Google's BigTable
george.james
 
Report from DEVCON 2008
Report from DEVCON 2008Report from DEVCON 2008
Report from DEVCON 2008
george.james
 
Michelle's Wallpaper
Michelle's WallpaperMichelle's Wallpaper
Michelle's Wallpaper
george.james
 
Beyond The MVC
Beyond The MVCBeyond The MVC
Beyond The MVC
george.james
 
Amazon S3 and EC2
Amazon S3 and EC2Amazon S3 and EC2
Amazon S3 and EC2
george.james
 
FIS-PIP™ – A high end database application development platform
FIS-PIP™ – A high end database application development platformFIS-PIP™ – A high end database application development platform
FIS-PIP™ – A high end database application development platform
george.james
 
Web Design and Programming
Web Design and ProgrammingWeb Design and Programming
Web Design and Programming
george.james
 
Querying the Web
Querying the WebQuerying the Web
Querying the Web
george.james
 
Mission-critical Ajax: Making Test Ordering Easier and Faster at Quest Diagno...
Mission-critical Ajax:Making Test Ordering Easier and Faster at Quest Diagno...Mission-critical Ajax:Making Test Ordering Easier and Faster at Quest Diagno...
Mission-critical Ajax: Making Test Ordering Easier and Faster at Quest Diagno...
george.james
 
Fosdem 2010 GT.M and OpenStreetMap
Fosdem 2010 GT.M and OpenStreetMapFosdem 2010 GT.M and OpenStreetMap
Fosdem 2010 GT.M and OpenStreetMap
george.james
 
Lost In The Clouds
Lost In The CloudsLost In The Clouds
Lost In The Clouds
george.james
 
On a cloudy day you can scale forever
On a cloudy day you can scale foreverOn a cloudy day you can scale forever
On a cloudy day you can scale forever
george.james
 
Bad Light Stops Play
Bad Light Stops PlayBad Light Stops Play
Bad Light Stops Play
george.james
 
Securing The Cloud
Securing The CloudSecuring The Cloud
Securing The Cloud
george.james
 
Out Of The Slipstream Proposal
Out Of The Slipstream ProposalOut Of The Slipstream Proposal
Out Of The Slipstream Proposal
george.james
 
Lightning In The Clouds
Lightning In The CloudsLightning In The Clouds
Lightning In The Clouds
george.james
 
Lost In The Clouds
Lost In The CloudsLost In The Clouds
Lost In The Clouds
george.james
 
Mumps the Internet scale database
Mumps the Internet scale databaseMumps the Internet scale database
Mumps the Internet scale database
george.james
 
Web Development Environments: Choose the best or go with the rest
Web Development Environments:  Choose the best or go with the restWeb Development Environments:  Choose the best or go with the rest
Web Development Environments: Choose the best or go with the rest
george.james
 
Report from DEVCON 2008
Report from DEVCON 2008Report from DEVCON 2008
Report from DEVCON 2008
george.james
 
Michelle's Wallpaper
Michelle's WallpaperMichelle's Wallpaper
Michelle's Wallpaper
george.james
 
FIS-PIP™ – A high end database application development platform
FIS-PIP™ – A high end database application development platformFIS-PIP™ – A high end database application development platform
FIS-PIP™ – A high end database application development platform
george.james
 
Web Design and Programming
Web Design and ProgrammingWeb Design and Programming
Web Design and Programming
george.james
 
Mission-critical Ajax: Making Test Ordering Easier and Faster at Quest Diagno...
Mission-critical Ajax:Making Test Ordering Easier and Faster at Quest Diagno...Mission-critical Ajax:Making Test Ordering Easier and Faster at Quest Diagno...
Mission-critical Ajax: Making Test Ordering Easier and Faster at Quest Diagno...
george.james
 

Recently uploaded (20)

Adverse event following immunization (AEFI).pptx
Adverse event following immunization (AEFI).pptxAdverse event following immunization (AEFI).pptx
Adverse event following immunization (AEFI).pptx
Dr. Koppala R.V.S. Chaitanya
 
Normal distribution and Z score Test for post graduate and undergraduate stu...
Normal distribution and Z score Test  for post graduate and undergraduate stu...Normal distribution and Z score Test  for post graduate and undergraduate stu...
Normal distribution and Z score Test for post graduate and undergraduate stu...
Tauseef Jawaid
 
PELVIC LYMPH NODES TARGET DELINEATION Dr Syed Aman.pptx
PELVIC LYMPH NODES TARGET DELINEATION Dr Syed Aman.pptxPELVIC LYMPH NODES TARGET DELINEATION Dr Syed Aman.pptx
PELVIC LYMPH NODES TARGET DELINEATION Dr Syed Aman.pptx
Syed Aman
 
Taking the Lead in Timely Diagnosis of AD: Incorporating Biomarkers Into Rout...
Taking the Lead in Timely Diagnosis of AD: Incorporating Biomarkers Into Rout...Taking the Lead in Timely Diagnosis of AD: Incorporating Biomarkers Into Rout...
Taking the Lead in Timely Diagnosis of AD: Incorporating Biomarkers Into Rout...
PVI, PeerView Institute for Medical Education
 
Lung Cancer: Artificial Intelligence, Synergetics, Complex System Analysis, B...
Lung Cancer: Artificial Intelligence, Synergetics, Complex System Analysis, B...Lung Cancer: Artificial Intelligence, Synergetics, Complex System Analysis, B...
Lung Cancer: Artificial Intelligence, Synergetics, Complex System Analysis, B...
Oleg Kshivets
 
Meeting dissolution requirements M.Pharmacy sem 2nd biopharmaceutics &pharmac...
Meeting dissolution requirements M.Pharmacy sem 2nd biopharmaceutics &pharmac...Meeting dissolution requirements M.Pharmacy sem 2nd biopharmaceutics &pharmac...
Meeting dissolution requirements M.Pharmacy sem 2nd biopharmaceutics &pharmac...
Swami ramanand teerth marathwada university
 
Analgesia system & Abnormalities of Pain_AntiCopy.pdf
Analgesia system & Abnormalities of Pain_AntiCopy.pdfAnalgesia system & Abnormalities of Pain_AntiCopy.pdf
Analgesia system & Abnormalities of Pain_AntiCopy.pdf
MedicoseAcademics
 
PLEURA & IT'S RECESSES -Prof.Dr.N.Mugunthan.pdf
PLEURA & IT'S RECESSES -Prof.Dr.N.Mugunthan.pdfPLEURA & IT'S RECESSES -Prof.Dr.N.Mugunthan.pdf
PLEURA & IT'S RECESSES -Prof.Dr.N.Mugunthan.pdf
Kanyakumari Medical Mission Research Center, Muttom
 
WOUND HEALING IN PERIODONTOLOGY - 3.pptx
WOUND HEALING IN PERIODONTOLOGY - 3.pptxWOUND HEALING IN PERIODONTOLOGY - 3.pptx
WOUND HEALING IN PERIODONTOLOGY - 3.pptx
tarunprakash1904
 
Information Resources In Pharmacovigilance.pptx
Information Resources In Pharmacovigilance.pptxInformation Resources In Pharmacovigilance.pptx
Information Resources In Pharmacovigilance.pptx
Dr. Koppala R.V.S. Chaitanya
 
Ophthalmological notes for dental students
Ophthalmological notes for dental studentsOphthalmological notes for dental students
Ophthalmological notes for dental students
KafrELShiekh University
 
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdfSubconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
wahbikhalidali
 
Defining and Delivering Person-Centric HIV Care in Key Populations
Defining and Delivering Person-Centric HIV Care in Key PopulationsDefining and Delivering Person-Centric HIV Care in Key Populations
Defining and Delivering Person-Centric HIV Care in Key Populations
PVI, PeerView Institute for Medical Education
 
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdfSubconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
wahbikhalidali
 
PLEURAL EFFUSION By Dr. Kritika Somani..
PLEURAL EFFUSION By Dr. Kritika Somani..PLEURAL EFFUSION By Dr. Kritika Somani..
PLEURAL EFFUSION By Dr. Kritika Somani..
KritikaSomani1
 
A comparative study of onlay versus sublay mesh repair in the surgical manage...
A comparative study of onlay versus sublay mesh repair in the surgical manage...A comparative study of onlay versus sublay mesh repair in the surgical manage...
A comparative study of onlay versus sublay mesh repair in the surgical manage...
Sona Thesis Consultancy
 
SPECIFIC ETHICAL ISSUES, FFP, AUTHORSHIP,CONFLICT OF INTEREST.pptx
SPECIFIC ETHICAL ISSUES, FFP, AUTHORSHIP,CONFLICT OF INTEREST.pptxSPECIFIC ETHICAL ISSUES, FFP, AUTHORSHIP,CONFLICT OF INTEREST.pptx
SPECIFIC ETHICAL ISSUES, FFP, AUTHORSHIP,CONFLICT OF INTEREST.pptx
SanskritiUpadhyay5
 
The Physiology of Central Nervous System - Sensory Pathways
The Physiology of Central Nervous System - Sensory PathwaysThe Physiology of Central Nervous System - Sensory Pathways
The Physiology of Central Nervous System - Sensory Pathways
MedicoseAcademics
 
Gastric Cancer: Artificial Intelligence, Synergetics, Complex System Analysis...
Gastric Cancer: Artificial Intelligence, Synergetics, Complex System Analysis...Gastric Cancer: Artificial Intelligence, Synergetics, Complex System Analysis...
Gastric Cancer: Artificial Intelligence, Synergetics, Complex System Analysis...
Oleg Kshivets
 
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptxBIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
drnidhimnd
 
Normal distribution and Z score Test for post graduate and undergraduate stu...
Normal distribution and Z score Test  for post graduate and undergraduate stu...Normal distribution and Z score Test  for post graduate and undergraduate stu...
Normal distribution and Z score Test for post graduate and undergraduate stu...
Tauseef Jawaid
 
PELVIC LYMPH NODES TARGET DELINEATION Dr Syed Aman.pptx
PELVIC LYMPH NODES TARGET DELINEATION Dr Syed Aman.pptxPELVIC LYMPH NODES TARGET DELINEATION Dr Syed Aman.pptx
PELVIC LYMPH NODES TARGET DELINEATION Dr Syed Aman.pptx
Syed Aman
 
Lung Cancer: Artificial Intelligence, Synergetics, Complex System Analysis, B...
Lung Cancer: Artificial Intelligence, Synergetics, Complex System Analysis, B...Lung Cancer: Artificial Intelligence, Synergetics, Complex System Analysis, B...
Lung Cancer: Artificial Intelligence, Synergetics, Complex System Analysis, B...
Oleg Kshivets
 
Analgesia system & Abnormalities of Pain_AntiCopy.pdf
Analgesia system & Abnormalities of Pain_AntiCopy.pdfAnalgesia system & Abnormalities of Pain_AntiCopy.pdf
Analgesia system & Abnormalities of Pain_AntiCopy.pdf
MedicoseAcademics
 
WOUND HEALING IN PERIODONTOLOGY - 3.pptx
WOUND HEALING IN PERIODONTOLOGY - 3.pptxWOUND HEALING IN PERIODONTOLOGY - 3.pptx
WOUND HEALING IN PERIODONTOLOGY - 3.pptx
tarunprakash1904
 
Ophthalmological notes for dental students
Ophthalmological notes for dental studentsOphthalmological notes for dental students
Ophthalmological notes for dental students
KafrELShiekh University
 
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdfSubconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
wahbikhalidali
 
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdfSubconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
wahbikhalidali
 
PLEURAL EFFUSION By Dr. Kritika Somani..
PLEURAL EFFUSION By Dr. Kritika Somani..PLEURAL EFFUSION By Dr. Kritika Somani..
PLEURAL EFFUSION By Dr. Kritika Somani..
KritikaSomani1
 
A comparative study of onlay versus sublay mesh repair in the surgical manage...
A comparative study of onlay versus sublay mesh repair in the surgical manage...A comparative study of onlay versus sublay mesh repair in the surgical manage...
A comparative study of onlay versus sublay mesh repair in the surgical manage...
Sona Thesis Consultancy
 
SPECIFIC ETHICAL ISSUES, FFP, AUTHORSHIP,CONFLICT OF INTEREST.pptx
SPECIFIC ETHICAL ISSUES, FFP, AUTHORSHIP,CONFLICT OF INTEREST.pptxSPECIFIC ETHICAL ISSUES, FFP, AUTHORSHIP,CONFLICT OF INTEREST.pptx
SPECIFIC ETHICAL ISSUES, FFP, AUTHORSHIP,CONFLICT OF INTEREST.pptx
SanskritiUpadhyay5
 
The Physiology of Central Nervous System - Sensory Pathways
The Physiology of Central Nervous System - Sensory PathwaysThe Physiology of Central Nervous System - Sensory Pathways
The Physiology of Central Nervous System - Sensory Pathways
MedicoseAcademics
 
Gastric Cancer: Artificial Intelligence, Synergetics, Complex System Analysis...
Gastric Cancer: Artificial Intelligence, Synergetics, Complex System Analysis...Gastric Cancer: Artificial Intelligence, Synergetics, Complex System Analysis...
Gastric Cancer: Artificial Intelligence, Synergetics, Complex System Analysis...
Oleg Kshivets
 
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptxBIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
drnidhimnd
 

The experiences of migrating a large scale, high performance healthcare network

  • 1. The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare
  • 2. In the next half hour… Partners Healthcare System overview Caché platform architecture & metrics The need to migrate Phased migration approach Benchmark testing and results Discoveries and production enhancements
  • 3. Partners Healthcare System Founded in 1994 Brigham & Women’s Hospital Massachusetts General Hospital Now includes: Community physician network (1200 + 3500 MD’s) PCHi 3 community hospitals 2 rehab hospitals 3 specialty institutions Enterprise-wide Information Systems 1100 employees Annual budget FY05 approximately $160 million
  • 4. Anchor Hospitals & Airport BWH MGH Logan Airport 10 km 6 km
  • 5. Acute Care Hospitals MGH BWH Newton- Wellesley Community Physician Practices
  • 6. Partners Domain Devices Internet 12,000 Printers 32,000 Desktops Firewall ~30,000 other devices 1,450 Servers Closely Managed Assumed Managed
  • 8. Enterprise Integration Over 30% are to and from Caché database Change from prior year Daily Average Est. Annual Transactions # of Interfaces 196 170 192 167 37% 4,659,035 1,330,962,017 2007 40% 3,399,211 1,240,712,044 2006 45% 2,431,917 887,649,802 2005 1,673,515 610,833,080 2004
  • 12. Database Utilization Average Database References per day in Billions
  • 13. The Need to Migrate - Availability Monthly Downtime Current State Business need
  • 14. Additional Business Requirements Increase availability and reliability Decrease database risk from 5 single points of failure More robust hardware and OS Many less servers and OS instances to manage Clustering and automated failover Reduce monthly maintenance needs, updates once or twice per year -------------------------------------------------------- Improve Performance 64 bit OS, more memory for cache Caché 5.0.20 to Caché 2008.1, significantly improved ECP performance Increase Scalability 91 Terabytes available on EMC SAN DMX3 On-demand addition of processor cores
  • 15. Caché Migration Decision Making Process Only considered first tier vendors and support (IBM, HP) HP assumed much more risk with Professional Services Existing HP business yields more leverage & visibility with regional office More headroom in HP configuration Price was not a distinguishing factor
  • 16. Phased migration approach Proof of Concept (benchmark testing) Completed 10/15/07 Phase 1 – Database tier 4 of 5 servers migrated, anticipated completion 4/14/08 Phase 2 – Application tier Big Bang migration 12/14/08 Phase 3 – Disaster Recovery January 2009
  • 18. Database Benchmark Load Testing Results Goals Simulate current Production user counts & transaction loads Verify support for load increases up to 300% Benchmark Environment Isolated LAN, new DMX3 SAN 20 new Windows blade servers (10 app servers, 10 script ‘players’) Scripts for 8 apps (represent heaviest use, Web/Telnet/VB apps) 2 batch jobs (screensaver simulation, NullGen LMR functions) Conclusions Able to simulate production load, 1.5x and 3x load 2 HP rx8640 can handle growth projections 0.66 0.15 0.32 LMR avg Caché app time (in sec.) 40,000 40,000 11,806 LMR transactions (5 min. period) 135,000 30,000 35,000 Database Global Refs / sec. Benchmark full script load Benchmark “paced” script load Production peak (8/21, 11:20 am) Metric
  • 19. Design and Configuration Considerations Database configuration simulation testing 1 to 5 Caché database instances were assessed 1 vs. 5 ECP channels per Caché instance were assessed Number of active cores were accessed (4 active, 2 reserved) Results and unexpected discoveries Identify 5 Caché database instance as optimal design configuration Journal synch bottleneck the biggest issue High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance) Maintain same data distribution across 5 DB instances Determine 1 ECP channel per instance optimal Additional channels did not improve throughput, still have only 1 Journal Deamon
  • 20. Benchmark Discoveries led to Production Improvements References to Undefined globals using $Data and $Get  These commands require network round trip Use of $increment Each call to $I requires network round trip Excessive use of Cache locks Forces more than 1 round trip Use of large strings Strings that require more than 3900–4000 bytes to represent the string value are big strings and never cached on the ECP client. Lesson Learned - Each trip to the database server results in overhead caused by a Journal Synch.  Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions .
  • 21. 75% reduction in long running transaction
  • 23. Monthly Average Caché Web Transaction Time
  • 24. Application Models Old New Browser client Web server Cache Cache VB client .Net server Cache Cache .Net client Browser client Web server Cache Web Services Browser client .Net client Scalability/Connection pooling, robustness/error handling, Vism Managed Obj. Vism.ocx Managed Obj. Cache Web Services WebLink
  • 25. The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare