SlideShare a Scribd company logo
The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare
In the next half hour… Partners Healthcare System overview Caché platform architecture & metrics The need to migrate Phased migration approach Benchmark testing and results Discoveries and production enhancements Post database migration results Future milestones
Partners Healthcare System Founded in 1994  Brigham & Women’s Hospital  Massachusetts General Hospital Now includes: Community physician network (1200 + 3500 MD’s) PCHi 3 community hospitals 2 rehab hospitals 3 specialty institutions  Enterprise-wide Information Systems 1100 employees Annual budget FY05 approximately $160 million
Anchor Hospitals & Airport BWH MGH Logan Airport 10 km 6 km
Acute Care Hospitals MGH BWH Newton- Wellesley Community Physician Practices
Partners Domain Devices Internet 12,000 Printers 32,000 Desktops Firewall ~30,000 other devices 1,450 Servers Closely Managed Assumed Managed
Current Mixed-Mode  Production Architecture
Enterprise Integration Over 30% are to and from Caché database Change from prior year Daily  Average Est. Annual Transactions # of Interfaces 196 170 192 167 37% 4,659,035 1,330,962,017 2007 40% 3,399,211 1,240,712,044 2006 45% 2,431,917 887,649,802 2005 1,673,515 610,833,080 2004
Integration Components
Annual Database Growth Rate
Database Utilization Average Database References per day in Billions
The Need to Migrate - Availability Monthly Downtime Current  State Business  need
Additional Business Requirements Increase availability and reliability Decrease database risk from 5 single points of failure More robust hardware and OS Many less servers and OS instances to manage Clustering and automated failover  Reduce monthly maintenance needs, updates once or twice per year -------------------------------------------------------- Improve Performance  64 bit OS, more memory for cache Caché 5.0.20 to Caché 2008.1, significantly improved  ECP performance Increase Scalability 91 Terabytes available on EMC SAN DMX3 On-demand addition of processor cores
Caché Migration Decision Making Process Only considered first tier vendors and support  (IBM, HP) HP assumed much more risk with Professional  Services Existing HP business yields more leverage & visibility with regional office More headroom in HP configuration Price was not a distinguishing factor
Phased migration approach Proof of Concept (benchmark testing) Completed 10/15/07 Phase 1 – Database tier Completed 4/14/08 Phase 2 – Application tier Big Bang migration 12/14/08 (includes Cache 2008.1 UNIX platform upgrade) Phase 3 – Disaster Recovery Now implemented as part of Phase 2, leveraging HP Blade servers
UNIX DataBase Tier Benchmark Environment
UNIX Application Tier Benchmark Environment
Benchmark Load Testing Results Goals  Simulate current Production user counts & transaction loads Verify support for load increases up to 3x Benchmark Environment Isolated LAN, new DMX3 SAN 20 new Windows blade servers (10 app servers, 10 script ‘players’) Scripts for 8 apps (represent heaviest use, Web/Telnet/VB apps) 2 batch jobs (screensaver simulation, NullGen LMR functions) Conclusions Able to simulate production load, 1.5x and 3x load 2 HP rx8640 can handle growth projections 0.66 0.15 0.32 LMR avg Caché app time (in sec.) 40,000 40,000 11,806 LMR transactions (5 min. period) 135,000 30,000 35,000 Database Global Refs / sec. Benchmark full script load Benchmark “paced” script load Production peak (8/21, 11:20 am) Metric
Design and Configuration Considerations Database configuration simulation testing 1 to 5 Caché database instances were assessed 1 vs. 5 ECP channels per Caché instance were assessed Number of active cores were accessed (4 active, 2  reserved) Results and unexpected discoveries Identify 5 Caché database instance as optimal design configuration Maintain same data distribution across 5 DB instances Journal synch bottleneck the biggest issue  High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance) Determine 1 ECP channel per instance optimal Additional channels did not improve throughput, still have only 1 Journal Deamon
Benchmark Discoveries Led to Production Improvements (the rules have changed) References to  Undefined  globals using $Data and $Get  These commands require network round trip Use of $increment  Each call to $I requires network round trip Excessive use of Cache locks  Forces more than 1 round trip Use of large strings  Strings that require more than 3900–4000 bytes to represent the string value are big strings and never cached on the ECP client.   Lesson Learned -  Each trip to the database server results in overhead caused by a Journal Synch.  Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions.
Post UNIX DB migration - 50% Reduction in Outlier Transaction
Monthly Average Caché Web Transaction Time
Unprecedented Growth - The LMR
Phased/Parallel Migration Approach
Parallel Migration Testing and Certification Effort DEV VC/m DEV QA PROD QA auto copy on promote promote check-out & check-in Developer test Analyst cert Production release promote auto copy on check-in Caché Environment 5.0 (WIN) 2008.1 (UNIX)
Application Models Old   New Browser  client Web server Cache Cache VB client .Net server Cache Cache .Net client Browser  client Web server Cache Web Services Browser  client .Net client Scalability/Connection pooling, robustness/error handling, Vism Managed Provider Vism.ocx (MServices) Managed Provider Cache Web Services WebLink
The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare
Ad

More Related Content

What's hot (20)

ScalabilityAvailability
ScalabilityAvailabilityScalabilityAvailability
ScalabilityAvailability
webuploader
 
Server load balancer ppt
Server load balancer pptServer load balancer ppt
Server load balancer ppt
Shilpi Tandon
 
Load Balancing Server
Load Balancing ServerLoad Balancing Server
Load Balancing Server
abhishek16pradhan
 
saito_porcupine
saito_porcupinesaito_porcupine
saito_porcupine
webuploader
 
Load Balancing
Load BalancingLoad Balancing
Load Balancing
nashniv
 
Performance and Scalability Tuning
Performance and Scalability TuningPerformance and Scalability Tuning
Performance and Scalability Tuning
Andres March
 
Load Balancing from the Cloud - Layer 7 Aware Solution
Load Balancing from the Cloud - Layer 7 Aware SolutionLoad Balancing from the Cloud - Layer 7 Aware Solution
Load Balancing from the Cloud - Layer 7 Aware Solution
Imperva Incapsula
 
top 15 easy performance optimizations for Veeam
top 15 easy performance optimizations for Veeamtop 15 easy performance optimizations for Veeam
top 15 easy performance optimizations for Veeam
Tanawit Chansuchai
 
Training Webinar: Enterprise application performance with distributed caching
Training Webinar: Enterprise application performance with distributed cachingTraining Webinar: Enterprise application performance with distributed caching
Training Webinar: Enterprise application performance with distributed caching
OutSystems
 
Veeam Availability Console What's News at Veeam ON Thailand
Veeam Availability Console What's News at Veeam ON ThailandVeeam Availability Console What's News at Veeam ON Thailand
Veeam Availability Console What's News at Veeam ON Thailand
Tanawit Chansuchai
 
Webinar Slides: Real-Time Replication vs. ETL - How Analytics Requires New Te...
Webinar Slides: Real-Time Replication vs. ETL - How Analytics Requires New Te...Webinar Slides: Real-Time Replication vs. ETL - How Analytics Requires New Te...
Webinar Slides: Real-Time Replication vs. ETL - How Analytics Requires New Te...
Continuent
 
Access Db to SQL Server Migration
Access Db to SQL Server MigrationAccess Db to SQL Server Migration
Access Db to SQL Server Migration
James Igoe
 
QA. Load Testing
QA. Load TestingQA. Load Testing
QA. Load Testing
Alex Galkin
 
Load Balancing And Yahoo!
Load Balancing And Yahoo!Load Balancing And Yahoo!
Load Balancing And Yahoo!
TwinVasion
 
SFDCP – sfdc project
SFDCP – sfdc projectSFDCP – sfdc project
SFDCP – sfdc project
Peter, Ivan Orlik
 
Training Webinar: Detect Performance Bottlenecks of Applications
Training Webinar: Detect Performance Bottlenecks of ApplicationsTraining Webinar: Detect Performance Bottlenecks of Applications
Training Webinar: Detect Performance Bottlenecks of Applications
OutSystems
 
Better performances with HTTP/2
Better performances with HTTP/2Better performances with HTTP/2
Better performances with HTTP/2
Thomas Segismont
 
Clustering - october 2006
Clustering  - october 2006Clustering  - october 2006
Clustering - october 2006
achraf_ing
 
Hyper V In Windows Server 2008 R2.Son Vu
Hyper V In Windows Server 2008 R2.Son VuHyper V In Windows Server 2008 R2.Son Vu
Hyper V In Windows Server 2008 R2.Son Vu
vncson
 
Optimizing performance
Optimizing performanceOptimizing performance
Optimizing performance
Zend by Rogue Wave Software
 
ScalabilityAvailability
ScalabilityAvailabilityScalabilityAvailability
ScalabilityAvailability
webuploader
 
Server load balancer ppt
Server load balancer pptServer load balancer ppt
Server load balancer ppt
Shilpi Tandon
 
Load Balancing
Load BalancingLoad Balancing
Load Balancing
nashniv
 
Performance and Scalability Tuning
Performance and Scalability TuningPerformance and Scalability Tuning
Performance and Scalability Tuning
Andres March
 
Load Balancing from the Cloud - Layer 7 Aware Solution
Load Balancing from the Cloud - Layer 7 Aware SolutionLoad Balancing from the Cloud - Layer 7 Aware Solution
Load Balancing from the Cloud - Layer 7 Aware Solution
Imperva Incapsula
 
top 15 easy performance optimizations for Veeam
top 15 easy performance optimizations for Veeamtop 15 easy performance optimizations for Veeam
top 15 easy performance optimizations for Veeam
Tanawit Chansuchai
 
Training Webinar: Enterprise application performance with distributed caching
Training Webinar: Enterprise application performance with distributed cachingTraining Webinar: Enterprise application performance with distributed caching
Training Webinar: Enterprise application performance with distributed caching
OutSystems
 
Veeam Availability Console What's News at Veeam ON Thailand
Veeam Availability Console What's News at Veeam ON ThailandVeeam Availability Console What's News at Veeam ON Thailand
Veeam Availability Console What's News at Veeam ON Thailand
Tanawit Chansuchai
 
Webinar Slides: Real-Time Replication vs. ETL - How Analytics Requires New Te...
Webinar Slides: Real-Time Replication vs. ETL - How Analytics Requires New Te...Webinar Slides: Real-Time Replication vs. ETL - How Analytics Requires New Te...
Webinar Slides: Real-Time Replication vs. ETL - How Analytics Requires New Te...
Continuent
 
Access Db to SQL Server Migration
Access Db to SQL Server MigrationAccess Db to SQL Server Migration
Access Db to SQL Server Migration
James Igoe
 
QA. Load Testing
QA. Load TestingQA. Load Testing
QA. Load Testing
Alex Galkin
 
Load Balancing And Yahoo!
Load Balancing And Yahoo!Load Balancing And Yahoo!
Load Balancing And Yahoo!
TwinVasion
 
Training Webinar: Detect Performance Bottlenecks of Applications
Training Webinar: Detect Performance Bottlenecks of ApplicationsTraining Webinar: Detect Performance Bottlenecks of Applications
Training Webinar: Detect Performance Bottlenecks of Applications
OutSystems
 
Better performances with HTTP/2
Better performances with HTTP/2Better performances with HTTP/2
Better performances with HTTP/2
Thomas Segismont
 
Clustering - october 2006
Clustering  - october 2006Clustering  - october 2006
Clustering - october 2006
achraf_ing
 
Hyper V In Windows Server 2008 R2.Son Vu
Hyper V In Windows Server 2008 R2.Son VuHyper V In Windows Server 2008 R2.Son Vu
Hyper V In Windows Server 2008 R2.Son Vu
vncson
 

Viewers also liked (9)

Bloody Sunday Photo Album
Bloody  Sunday  Photo  AlbumBloody  Sunday  Photo  Album
Bloody Sunday Photo Album
irelandinschools
 
Lost In The Clouds
Lost In The CloudsLost In The Clouds
Lost In The Clouds
george.james
 
Michelle's Wallpaper
Michelle's WallpaperMichelle's Wallpaper
Michelle's Wallpaper
george.james
 
Freshman Orientation
Freshman OrientationFreshman Orientation
Freshman Orientation
DanaHawes
 
Out Of The Slipstream Proposal
Out Of The Slipstream ProposalOut Of The Slipstream Proposal
Out Of The Slipstream Proposal
george.james
 
Beyond The MVC
Beyond The MVCBeyond The MVC
Beyond The MVC
george.james
 
Template[1]
Template[1]Template[1]
Template[1]
vflores
 
The Bridge
The BridgeThe Bridge
The Bridge
nppsagar
 
Bmw Pp
Bmw PpBmw Pp
Bmw Pp
guest9f30a4
 
Lost In The Clouds
Lost In The CloudsLost In The Clouds
Lost In The Clouds
george.james
 
Michelle's Wallpaper
Michelle's WallpaperMichelle's Wallpaper
Michelle's Wallpaper
george.james
 
Freshman Orientation
Freshman OrientationFreshman Orientation
Freshman Orientation
DanaHawes
 
Out Of The Slipstream Proposal
Out Of The Slipstream ProposalOut Of The Slipstream Proposal
Out Of The Slipstream Proposal
george.james
 
Template[1]
Template[1]Template[1]
Template[1]
vflores
 
The Bridge
The BridgeThe Bridge
The Bridge
nppsagar
 
Ad

Similar to The experiences of migrating a large scale, high performance healthcare network (20)

The experiences of migrating a large scale, high performance healthcare network
The experiences of migrating a large scale, high performance healthcare networkThe experiences of migrating a large scale, high performance healthcare network
The experiences of migrating a large scale, high performance healthcare network
george.james
 
Hpe service virtualization 3.8 what's new chicago adm
Hpe service virtualization 3.8 what's new chicago admHpe service virtualization 3.8 what's new chicago adm
Hpe service virtualization 3.8 what's new chicago adm
Jeffrey Nunn
 
Introduction To Cloud Computing
Introduction To Cloud ComputingIntroduction To Cloud Computing
Introduction To Cloud Computing
Rinat Shagisultanov
 
Real Time Network Monitoring System
Real  Time  Network  Monitoring  SystemReal  Time  Network  Monitoring  System
Real Time Network Monitoring System
Girish Naik
 
3 Hyper V
3 Hyper V3 Hyper V
3 Hyper V
johnbakerMS
 
Virtualising Tier 1 Apps
Virtualising Tier 1 AppsVirtualising Tier 1 Apps
Virtualising Tier 1 Apps
Iwan Rahabok
 
T3 Consortium's Performance Center of Excellence
T3 Consortium's Performance Center of ExcellenceT3 Consortium's Performance Center of Excellence
T3 Consortium's Performance Center of Excellence
veehikle
 
Partners Enterprise Caché Unix Migration Public
Partners Enterprise Caché Unix Migration PublicPartners Enterprise Caché Unix Migration Public
Partners Enterprise Caché Unix Migration Public
walterhalvorsen
 
Asynchronous Mobile Web Services:
Asynchronous Mobile Web Services: Asynchronous Mobile Web Services:
Asynchronous Mobile Web Services:
Dr. Fahad Aijaz
 
Case Study: How Cisco Gained Visibility into Network Utilization and Proacti...
Case Study:  How Cisco Gained Visibility into Network Utilization and Proacti...Case Study:  How Cisco Gained Visibility into Network Utilization and Proacti...
Case Study: How Cisco Gained Visibility into Network Utilization and Proacti...
CA Technologies
 
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache FlinkFlexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache Flink
DataWorks Summit
 
Webinar: Deploying the Combined Virtual and Physical Infrastructure
Webinar: Deploying the Combined Virtual and Physical InfrastructureWebinar: Deploying the Combined Virtual and Physical Infrastructure
Webinar: Deploying the Combined Virtual and Physical Infrastructure
Pepperweed Consulting
 
Java Abs Dynamic Server Replication
Java Abs   Dynamic Server ReplicationJava Abs   Dynamic Server Replication
Java Abs Dynamic Server Replication
ncct
 
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld
 
Si so product 1 day technical
Si so product 1 day technicalSi so product 1 day technical
Si so product 1 day technical
Bjørn Hell Larsen
 
Scvmm Technical Overview.Son Vu
Scvmm Technical Overview.Son VuScvmm Technical Overview.Son Vu
Scvmm Technical Overview.Son Vu
vncson
 
Sunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Sunx4450 Intel7460 GigaSpaces XAP Platform BenchmarkSunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Sunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Shay Hassidim
 
prezentációt
prezentációtprezentációt
prezentációt
webhostingguy
 
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Prolifics
 
Hp Connect 10 06 08 V5
Hp Connect 10 06 08 V5Hp Connect 10 06 08 V5
Hp Connect 10 06 08 V5
guestea711d0
 
The experiences of migrating a large scale, high performance healthcare network
The experiences of migrating a large scale, high performance healthcare networkThe experiences of migrating a large scale, high performance healthcare network
The experiences of migrating a large scale, high performance healthcare network
george.james
 
Hpe service virtualization 3.8 what's new chicago adm
Hpe service virtualization 3.8 what's new chicago admHpe service virtualization 3.8 what's new chicago adm
Hpe service virtualization 3.8 what's new chicago adm
Jeffrey Nunn
 
Real Time Network Monitoring System
Real  Time  Network  Monitoring  SystemReal  Time  Network  Monitoring  System
Real Time Network Monitoring System
Girish Naik
 
Virtualising Tier 1 Apps
Virtualising Tier 1 AppsVirtualising Tier 1 Apps
Virtualising Tier 1 Apps
Iwan Rahabok
 
T3 Consortium's Performance Center of Excellence
T3 Consortium's Performance Center of ExcellenceT3 Consortium's Performance Center of Excellence
T3 Consortium's Performance Center of Excellence
veehikle
 
Partners Enterprise Caché Unix Migration Public
Partners Enterprise Caché Unix Migration PublicPartners Enterprise Caché Unix Migration Public
Partners Enterprise Caché Unix Migration Public
walterhalvorsen
 
Asynchronous Mobile Web Services:
Asynchronous Mobile Web Services: Asynchronous Mobile Web Services:
Asynchronous Mobile Web Services:
Dr. Fahad Aijaz
 
Case Study: How Cisco Gained Visibility into Network Utilization and Proacti...
Case Study:  How Cisco Gained Visibility into Network Utilization and Proacti...Case Study:  How Cisco Gained Visibility into Network Utilization and Proacti...
Case Study: How Cisco Gained Visibility into Network Utilization and Proacti...
CA Technologies
 
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache FlinkFlexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache Flink
DataWorks Summit
 
Webinar: Deploying the Combined Virtual and Physical Infrastructure
Webinar: Deploying the Combined Virtual and Physical InfrastructureWebinar: Deploying the Combined Virtual and Physical Infrastructure
Webinar: Deploying the Combined Virtual and Physical Infrastructure
Pepperweed Consulting
 
Java Abs Dynamic Server Replication
Java Abs   Dynamic Server ReplicationJava Abs   Dynamic Server Replication
Java Abs Dynamic Server Replication
ncct
 
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
VMworld
 
Scvmm Technical Overview.Son Vu
Scvmm Technical Overview.Son VuScvmm Technical Overview.Son Vu
Scvmm Technical Overview.Son Vu
vncson
 
Sunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Sunx4450 Intel7460 GigaSpaces XAP Platform BenchmarkSunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Sunx4450 Intel7460 GigaSpaces XAP Platform Benchmark
Shay Hassidim
 
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...
Prolifics
 
Hp Connect 10 06 08 V5
Hp Connect 10 06 08 V5Hp Connect 10 06 08 V5
Hp Connect 10 06 08 V5
guestea711d0
 
Ad

More from george.james (20)

Fosdem 2010 GT.M and OpenStreetMap
Fosdem 2010 GT.M and OpenStreetMapFosdem 2010 GT.M and OpenStreetMap
Fosdem 2010 GT.M and OpenStreetMap
george.james
 
M/DB and M/DB:X
M/DB and M/DB:XM/DB and M/DB:X
M/DB and M/DB:X
george.james
 
Lost In The Clouds
Lost In The CloudsLost In The Clouds
Lost In The Clouds
george.james
 
On a cloudy day you can scale forever
On a cloudy day you can scale foreverOn a cloudy day you can scale forever
On a cloudy day you can scale forever
george.james
 
Bad Light Stops Play
Bad Light Stops PlayBad Light Stops Play
Bad Light Stops Play
george.james
 
Securing The Cloud
Securing The CloudSecuring The Cloud
Securing The Cloud
george.james
 
Lightning In The Clouds
Lightning In The CloudsLightning In The Clouds
Lightning In The Clouds
george.james
 
Mumps the Internet scale database
Mumps the Internet scale databaseMumps the Internet scale database
Mumps the Internet scale database
george.james
 
Web Development Environments: Choose the best or go with the rest
Web Development Environments:  Choose the best or go with the restWeb Development Environments:  Choose the best or go with the rest
Web Development Environments: Choose the best or go with the rest
george.james
 
Web Servers: Architecture and Security
Web Servers: Architecture and SecurityWeb Servers: Architecture and Security
Web Servers: Architecture and Security
george.james
 
Google's BigTable
Google's BigTableGoogle's BigTable
Google's BigTable
george.james
 
Report from DEVCON 2008
Report from DEVCON 2008Report from DEVCON 2008
Report from DEVCON 2008
george.james
 
Amazon S3 and EC2
Amazon S3 and EC2Amazon S3 and EC2
Amazon S3 and EC2
george.james
 
FIS-PIP™ – A high end database application development platform
FIS-PIP™ – A high end database application development platformFIS-PIP™ – A high end database application development platform
FIS-PIP™ – A high end database application development platform
george.james
 
Web Design and Programming
Web Design and ProgrammingWeb Design and Programming
Web Design and Programming
george.james
 
Querying the Web
Querying the WebQuerying the Web
Querying the Web
george.james
 
Mission-critical Ajax: Making Test Ordering Easier and Faster at Quest Diagno...
Mission-critical Ajax:Making Test Ordering Easier and Faster at Quest Diagno...Mission-critical Ajax:Making Test Ordering Easier and Faster at Quest Diagno...
Mission-critical Ajax: Making Test Ordering Easier and Faster at Quest Diagno...
george.james
 
Scalable Web Architectures and Infrastructure
Scalable Web Architectures and InfrastructureScalable Web Architectures and Infrastructure
Scalable Web Architectures and Infrastructure
george.james
 
Chris Munt Cherylin Fletcher State Of Play In Nhs
Chris Munt Cherylin Fletcher State Of Play In NhsChris Munt Cherylin Fletcher State Of Play In Nhs
Chris Munt Cherylin Fletcher State Of Play In Nhs
george.james
 
Rob Tweed :: Ajax and the Impact on Caché and Similar Technologies
Rob Tweed :: Ajax and the Impact on Caché and Similar TechnologiesRob Tweed :: Ajax and the Impact on Caché and Similar Technologies
Rob Tweed :: Ajax and the Impact on Caché and Similar Technologies
george.james
 
Fosdem 2010 GT.M and OpenStreetMap
Fosdem 2010 GT.M and OpenStreetMapFosdem 2010 GT.M and OpenStreetMap
Fosdem 2010 GT.M and OpenStreetMap
george.james
 
Lost In The Clouds
Lost In The CloudsLost In The Clouds
Lost In The Clouds
george.james
 
On a cloudy day you can scale forever
On a cloudy day you can scale foreverOn a cloudy day you can scale forever
On a cloudy day you can scale forever
george.james
 
Bad Light Stops Play
Bad Light Stops PlayBad Light Stops Play
Bad Light Stops Play
george.james
 
Securing The Cloud
Securing The CloudSecuring The Cloud
Securing The Cloud
george.james
 
Lightning In The Clouds
Lightning In The CloudsLightning In The Clouds
Lightning In The Clouds
george.james
 
Mumps the Internet scale database
Mumps the Internet scale databaseMumps the Internet scale database
Mumps the Internet scale database
george.james
 
Web Development Environments: Choose the best or go with the rest
Web Development Environments:  Choose the best or go with the restWeb Development Environments:  Choose the best or go with the rest
Web Development Environments: Choose the best or go with the rest
george.james
 
Web Servers: Architecture and Security
Web Servers: Architecture and SecurityWeb Servers: Architecture and Security
Web Servers: Architecture and Security
george.james
 
Report from DEVCON 2008
Report from DEVCON 2008Report from DEVCON 2008
Report from DEVCON 2008
george.james
 
FIS-PIP™ – A high end database application development platform
FIS-PIP™ – A high end database application development platformFIS-PIP™ – A high end database application development platform
FIS-PIP™ – A high end database application development platform
george.james
 
Web Design and Programming
Web Design and ProgrammingWeb Design and Programming
Web Design and Programming
george.james
 
Mission-critical Ajax: Making Test Ordering Easier and Faster at Quest Diagno...
Mission-critical Ajax:Making Test Ordering Easier and Faster at Quest Diagno...Mission-critical Ajax:Making Test Ordering Easier and Faster at Quest Diagno...
Mission-critical Ajax: Making Test Ordering Easier and Faster at Quest Diagno...
george.james
 
Scalable Web Architectures and Infrastructure
Scalable Web Architectures and InfrastructureScalable Web Architectures and Infrastructure
Scalable Web Architectures and Infrastructure
george.james
 
Chris Munt Cherylin Fletcher State Of Play In Nhs
Chris Munt Cherylin Fletcher State Of Play In NhsChris Munt Cherylin Fletcher State Of Play In Nhs
Chris Munt Cherylin Fletcher State Of Play In Nhs
george.james
 
Rob Tweed :: Ajax and the Impact on Caché and Similar Technologies
Rob Tweed :: Ajax and the Impact on Caché and Similar TechnologiesRob Tweed :: Ajax and the Impact on Caché and Similar Technologies
Rob Tweed :: Ajax and the Impact on Caché and Similar Technologies
george.james
 

Recently uploaded (20)

Subconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdfSubconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
wahbikhalidali
 
Role of Gene Therapy Neurological disorders
Role of Gene Therapy Neurological disordersRole of Gene Therapy Neurological disorders
Role of Gene Therapy Neurological disorders
riggdiana2
 
Pharmacology All Notes 505 Slides (2).pptx
Pharmacology All Notes 505 Slides (2).pptxPharmacology All Notes 505 Slides (2).pptx
Pharmacology All Notes 505 Slides (2).pptx
ssuseraed25f1
 
Common Male Sexual Problems | Best Sexologist in Patna, Bihar | Dr. Sunil Dubey
Common Male Sexual Problems | Best Sexologist in Patna, Bihar | Dr. Sunil DubeyCommon Male Sexual Problems | Best Sexologist in Patna, Bihar | Dr. Sunil Dubey
Common Male Sexual Problems | Best Sexologist in Patna, Bihar | Dr. Sunil Dubey
Sexologist Dr. Sunil Dubey - Dubey Clinic
 
Meeting dissolution requirements M.Pharmacy sem 2nd biopharmaceutics &pharmac...
Meeting dissolution requirements M.Pharmacy sem 2nd biopharmaceutics &pharmac...Meeting dissolution requirements M.Pharmacy sem 2nd biopharmaceutics &pharmac...
Meeting dissolution requirements M.Pharmacy sem 2nd biopharmaceutics &pharmac...
Swami ramanand teerth marathwada university
 
Introduction to the heart for pharm d I year.pptx
Introduction to the heart for pharm d I year.pptxIntroduction to the heart for pharm d I year.pptx
Introduction to the heart for pharm d I year.pptx
rekhapositivity
 
PLEURA & IT'S RECESSES -Prof.Dr.N.Mugunthan.pdf
PLEURA & IT'S RECESSES -Prof.Dr.N.Mugunthan.pdfPLEURA & IT'S RECESSES -Prof.Dr.N.Mugunthan.pdf
PLEURA & IT'S RECESSES -Prof.Dr.N.Mugunthan.pdf
Kanyakumari Medical Mission Research Center, Muttom
 
Physicochemical Nature of the Drug Formulation | Factors Affecting Drug Produ...
Physicochemical Nature of the Drug Formulation | Factors Affecting Drug Produ...Physicochemical Nature of the Drug Formulation | Factors Affecting Drug Produ...
Physicochemical Nature of the Drug Formulation | Factors Affecting Drug Produ...
Y Subhasish Singha
 
Terminologies of adverse medication related events , Regulatory terminologies.
Terminologies of adverse medication related events , Regulatory terminologies.Terminologies of adverse medication related events , Regulatory terminologies.
Terminologies of adverse medication related events , Regulatory terminologies.
Dr. Koppala R.V.S. Chaitanya
 
Diagnostics of Dental implant .scientific day 2025.pdf
Diagnostics of Dental implant .scientific day 2025.pdfDiagnostics of Dental implant .scientific day 2025.pdf
Diagnostics of Dental implant .scientific day 2025.pdf
Shimaa Hussein Kotb
 
Lecture chi squire. For Postgraduate and Undergraduate
Lecture chi squire. For Postgraduate and UndergraduateLecture chi squire. For Postgraduate and Undergraduate
Lecture chi squire. For Postgraduate and Undergraduate
Tauseef Jawaid
 
Normal distribution and Z score Test for post graduate and undergraduate stu...
Normal distribution and Z score Test  for post graduate and undergraduate stu...Normal distribution and Z score Test  for post graduate and undergraduate stu...
Normal distribution and Z score Test for post graduate and undergraduate stu...
Tauseef Jawaid
 
The Physiology of Central Nervous System - Sensory Pathways
The Physiology of Central Nervous System - Sensory PathwaysThe Physiology of Central Nervous System - Sensory Pathways
The Physiology of Central Nervous System - Sensory Pathways
MedicoseAcademics
 
Methods of Cancer diagnosis in Context of Radiotherapy
Methods of Cancer diagnosis in Context  of RadiotherapyMethods of Cancer diagnosis in Context  of Radiotherapy
Methods of Cancer diagnosis in Context of Radiotherapy
Saikat Roy
 
Severity and seriousness assessment: Pharmacovigilance aspects.
Severity and seriousness assessment: Pharmacovigilance aspects.Severity and seriousness assessment: Pharmacovigilance aspects.
Severity and seriousness assessment: Pharmacovigilance aspects.
Dr. Koppala R.V.S. Chaitanya
 
Adverse event following immunization (AEFI).pptx
Adverse event following immunization (AEFI).pptxAdverse event following immunization (AEFI).pptx
Adverse event following immunization (AEFI).pptx
Dr. Koppala R.V.S. Chaitanya
 
Cell Survival Curve (Radiobiology) Dr Syed Aman.pptx
Cell Survival Curve (Radiobiology) Dr Syed Aman.pptxCell Survival Curve (Radiobiology) Dr Syed Aman.pptx
Cell Survival Curve (Radiobiology) Dr Syed Aman.pptx
Syed Aman
 
Awake Craniotomy with endoscopic support, guided by intraoperative ultrasound...
Awake Craniotomy with endoscopic support, guided by intraoperative ultrasound...Awake Craniotomy with endoscopic support, guided by intraoperative ultrasound...
Awake Craniotomy with endoscopic support, guided by intraoperative ultrasound...
Dr. Damian Lastra Copello
 
Artificial Intelligence in Oncology: Transforming Cancer Carepptx
Artificial Intelligence in Oncology: Transforming Cancer CarepptxArtificial Intelligence in Oncology: Transforming Cancer Carepptx
Artificial Intelligence in Oncology: Transforming Cancer Carepptx
NEIGRIHMS, SHILLONG
 
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptxBIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
drnidhimnd
 
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdfSubconjunctival Hemorrhage Secondary to Pertussis.pdf
Subconjunctival Hemorrhage Secondary to Pertussis.pdf
wahbikhalidali
 
Role of Gene Therapy Neurological disorders
Role of Gene Therapy Neurological disordersRole of Gene Therapy Neurological disorders
Role of Gene Therapy Neurological disorders
riggdiana2
 
Pharmacology All Notes 505 Slides (2).pptx
Pharmacology All Notes 505 Slides (2).pptxPharmacology All Notes 505 Slides (2).pptx
Pharmacology All Notes 505 Slides (2).pptx
ssuseraed25f1
 
Common Male Sexual Problems | Best Sexologist in Patna, Bihar | Dr. Sunil Dubey
Common Male Sexual Problems | Best Sexologist in Patna, Bihar | Dr. Sunil DubeyCommon Male Sexual Problems | Best Sexologist in Patna, Bihar | Dr. Sunil Dubey
Common Male Sexual Problems | Best Sexologist in Patna, Bihar | Dr. Sunil Dubey
Sexologist Dr. Sunil Dubey - Dubey Clinic
 
Introduction to the heart for pharm d I year.pptx
Introduction to the heart for pharm d I year.pptxIntroduction to the heart for pharm d I year.pptx
Introduction to the heart for pharm d I year.pptx
rekhapositivity
 
Physicochemical Nature of the Drug Formulation | Factors Affecting Drug Produ...
Physicochemical Nature of the Drug Formulation | Factors Affecting Drug Produ...Physicochemical Nature of the Drug Formulation | Factors Affecting Drug Produ...
Physicochemical Nature of the Drug Formulation | Factors Affecting Drug Produ...
Y Subhasish Singha
 
Terminologies of adverse medication related events , Regulatory terminologies.
Terminologies of adverse medication related events , Regulatory terminologies.Terminologies of adverse medication related events , Regulatory terminologies.
Terminologies of adverse medication related events , Regulatory terminologies.
Dr. Koppala R.V.S. Chaitanya
 
Diagnostics of Dental implant .scientific day 2025.pdf
Diagnostics of Dental implant .scientific day 2025.pdfDiagnostics of Dental implant .scientific day 2025.pdf
Diagnostics of Dental implant .scientific day 2025.pdf
Shimaa Hussein Kotb
 
Lecture chi squire. For Postgraduate and Undergraduate
Lecture chi squire. For Postgraduate and UndergraduateLecture chi squire. For Postgraduate and Undergraduate
Lecture chi squire. For Postgraduate and Undergraduate
Tauseef Jawaid
 
Normal distribution and Z score Test for post graduate and undergraduate stu...
Normal distribution and Z score Test  for post graduate and undergraduate stu...Normal distribution and Z score Test  for post graduate and undergraduate stu...
Normal distribution and Z score Test for post graduate and undergraduate stu...
Tauseef Jawaid
 
The Physiology of Central Nervous System - Sensory Pathways
The Physiology of Central Nervous System - Sensory PathwaysThe Physiology of Central Nervous System - Sensory Pathways
The Physiology of Central Nervous System - Sensory Pathways
MedicoseAcademics
 
Methods of Cancer diagnosis in Context of Radiotherapy
Methods of Cancer diagnosis in Context  of RadiotherapyMethods of Cancer diagnosis in Context  of Radiotherapy
Methods of Cancer diagnosis in Context of Radiotherapy
Saikat Roy
 
Severity and seriousness assessment: Pharmacovigilance aspects.
Severity and seriousness assessment: Pharmacovigilance aspects.Severity and seriousness assessment: Pharmacovigilance aspects.
Severity and seriousness assessment: Pharmacovigilance aspects.
Dr. Koppala R.V.S. Chaitanya
 
Cell Survival Curve (Radiobiology) Dr Syed Aman.pptx
Cell Survival Curve (Radiobiology) Dr Syed Aman.pptxCell Survival Curve (Radiobiology) Dr Syed Aman.pptx
Cell Survival Curve (Radiobiology) Dr Syed Aman.pptx
Syed Aman
 
Awake Craniotomy with endoscopic support, guided by intraoperative ultrasound...
Awake Craniotomy with endoscopic support, guided by intraoperative ultrasound...Awake Craniotomy with endoscopic support, guided by intraoperative ultrasound...
Awake Craniotomy with endoscopic support, guided by intraoperative ultrasound...
Dr. Damian Lastra Copello
 
Artificial Intelligence in Oncology: Transforming Cancer Carepptx
Artificial Intelligence in Oncology: Transforming Cancer CarepptxArtificial Intelligence in Oncology: Transforming Cancer Carepptx
Artificial Intelligence in Oncology: Transforming Cancer Carepptx
NEIGRIHMS, SHILLONG
 
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptxBIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
BIOMECHANICS & KINESIOLOGY OF THEHIP COMPLEX.pptx
drnidhimnd
 

The experiences of migrating a large scale, high performance healthcare network

  • 1. The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare
  • 2. In the next half hour… Partners Healthcare System overview Caché platform architecture & metrics The need to migrate Phased migration approach Benchmark testing and results Discoveries and production enhancements Post database migration results Future milestones
  • 3. Partners Healthcare System Founded in 1994 Brigham & Women’s Hospital Massachusetts General Hospital Now includes: Community physician network (1200 + 3500 MD’s) PCHi 3 community hospitals 2 rehab hospitals 3 specialty institutions Enterprise-wide Information Systems 1100 employees Annual budget FY05 approximately $160 million
  • 4. Anchor Hospitals & Airport BWH MGH Logan Airport 10 km 6 km
  • 5. Acute Care Hospitals MGH BWH Newton- Wellesley Community Physician Practices
  • 6. Partners Domain Devices Internet 12,000 Printers 32,000 Desktops Firewall ~30,000 other devices 1,450 Servers Closely Managed Assumed Managed
  • 7. Current Mixed-Mode Production Architecture
  • 8. Enterprise Integration Over 30% are to and from Caché database Change from prior year Daily Average Est. Annual Transactions # of Interfaces 196 170 192 167 37% 4,659,035 1,330,962,017 2007 40% 3,399,211 1,240,712,044 2006 45% 2,431,917 887,649,802 2005 1,673,515 610,833,080 2004
  • 11. Database Utilization Average Database References per day in Billions
  • 12. The Need to Migrate - Availability Monthly Downtime Current State Business need
  • 13. Additional Business Requirements Increase availability and reliability Decrease database risk from 5 single points of failure More robust hardware and OS Many less servers and OS instances to manage Clustering and automated failover Reduce monthly maintenance needs, updates once or twice per year -------------------------------------------------------- Improve Performance 64 bit OS, more memory for cache Caché 5.0.20 to Caché 2008.1, significantly improved ECP performance Increase Scalability 91 Terabytes available on EMC SAN DMX3 On-demand addition of processor cores
  • 14. Caché Migration Decision Making Process Only considered first tier vendors and support (IBM, HP) HP assumed much more risk with Professional Services Existing HP business yields more leverage & visibility with regional office More headroom in HP configuration Price was not a distinguishing factor
  • 15. Phased migration approach Proof of Concept (benchmark testing) Completed 10/15/07 Phase 1 – Database tier Completed 4/14/08 Phase 2 – Application tier Big Bang migration 12/14/08 (includes Cache 2008.1 UNIX platform upgrade) Phase 3 – Disaster Recovery Now implemented as part of Phase 2, leveraging HP Blade servers
  • 16. UNIX DataBase Tier Benchmark Environment
  • 17. UNIX Application Tier Benchmark Environment
  • 18. Benchmark Load Testing Results Goals Simulate current Production user counts & transaction loads Verify support for load increases up to 3x Benchmark Environment Isolated LAN, new DMX3 SAN 20 new Windows blade servers (10 app servers, 10 script ‘players’) Scripts for 8 apps (represent heaviest use, Web/Telnet/VB apps) 2 batch jobs (screensaver simulation, NullGen LMR functions) Conclusions Able to simulate production load, 1.5x and 3x load 2 HP rx8640 can handle growth projections 0.66 0.15 0.32 LMR avg Caché app time (in sec.) 40,000 40,000 11,806 LMR transactions (5 min. period) 135,000 30,000 35,000 Database Global Refs / sec. Benchmark full script load Benchmark “paced” script load Production peak (8/21, 11:20 am) Metric
  • 19. Design and Configuration Considerations Database configuration simulation testing 1 to 5 Caché database instances were assessed 1 vs. 5 ECP channels per Caché instance were assessed Number of active cores were accessed (4 active, 2 reserved) Results and unexpected discoveries Identify 5 Caché database instance as optimal design configuration Maintain same data distribution across 5 DB instances Journal synch bottleneck the biggest issue High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance) Determine 1 ECP channel per instance optimal Additional channels did not improve throughput, still have only 1 Journal Deamon
  • 20. Benchmark Discoveries Led to Production Improvements (the rules have changed) References to Undefined globals using $Data and $Get  These commands require network round trip Use of $increment Each call to $I requires network round trip Excessive use of Cache locks Forces more than 1 round trip Use of large strings Strings that require more than 3900–4000 bytes to represent the string value are big strings and never cached on the ECP client. Lesson Learned - Each trip to the database server results in overhead caused by a Journal Synch.  Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions.
  • 21. Post UNIX DB migration - 50% Reduction in Outlier Transaction
  • 22. Monthly Average Caché Web Transaction Time
  • 25. Parallel Migration Testing and Certification Effort DEV VC/m DEV QA PROD QA auto copy on promote promote check-out & check-in Developer test Analyst cert Production release promote auto copy on check-in Caché Environment 5.0 (WIN) 2008.1 (UNIX)
  • 26. Application Models Old New Browser client Web server Cache Cache VB client .Net server Cache Cache .Net client Browser client Web server Cache Web Services Browser client .Net client Scalability/Connection pooling, robustness/error handling, Vism Managed Provider Vism.ocx (MServices) Managed Provider Cache Web Services WebLink
  • 27. The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare