Scaling paypal workloads with oracle rac ss
2 likes1,240 views
PayPal workloads scale efficiently with Oracle Real Application Clusters (RAC). Oracle RAC provides high availability, scalability, and performance for PayPal's workloads. Oracle RAC utilizes features like cache fusion, dynamic resource management, and fine-grained processes to optimize performance on modern CPU architectures. Oracle RAC also provides near-zero downtime for reconfiguration and patching through optimizations like recovery buddies. PayPal leverages over 250 Oracle RAC clusters with over 90 petabytes of database storage to handle its transaction processing and analytics workloads at scale with high performance.
![Optimize usage of OS Resources. -- Network
Cluster Communication is configured to use IPs from: GPnP
IP: 169.254.12.142 Subnet: 169.254.0.0
KSIPC Loopback IP addresses(OSD):
127.0.0.1
KSIPC Available Transports for cluster communication: UDP:TCP
KSIPC: Client: KCL Transport: UDP
KSIPC: Client: DLM Transport: UDP
KSIPC CAPABILITIES :MGA:IPCLW:GRPAM:TOPO:DLL:STATSFW:CRTRK
KSXP: ksxpsg_ipclwtrans: 2 UDP
cluster interconnect IPC version: [IPCLW over UDP(mode 3) ]
IPC Vendor 1 proto 2
Copyright © 2021, Oracle and/or its affiliates |](https://image.slidesharecdn.com/scalingpaypalworkloadswithoracleracss-210920153227/85/Scaling-paypal-workloads-with-oracle-rac-ss-22-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Scaling paypal workloads with oracle rac ss (20)
Ad
More from Anil Nair (10)
Ad
Recently uploaded (20)
Scaling paypal workloads with oracle rac ss
- 1. Scaling PayPal Workloads with Oracle RAC New Generation Oracle Real Application Clusters (RAC) Anil Nair ([email protected]) Sr. Principal Product Manager Oracle Real Application Clusters (RAC) & ASM @RACMasterPM http://www.linkedin.com/in/anil-nair-01960b6 http://www.slideshare.net/AnilNair27/
- 2. About Pramod
- 3. We are a trusted part of people’s financial lives and a partner to merchants in 200+ markets around the world Two decades ago, our founders invented payment technology to make buying and selling faster, secure, and easier; and put economic power where it belongs: In the hands of people About PayPal Our 390+ Million consumers can accept payments in > 100 currencies and interact with 20M+ Merchants across 19K+ corridors Almost 8000 PayPal team members provide support to our customers in over 20 languages
- 4. Click to add image “If you have not looked into the New Generation Oracle RAC 19c yet, you have not experienced what Oracle RAC can do for you.” https://www.oracle.com/technetwork/database/options/cluste ring/overview/new-generation-oracle-rac-5975370.pdf https://www.slideshare.net/AnilNair27/oracle-rac-new-generation Private Network Failover Private Network Copyright © 2021, Oracle and/or its affiliates |
- 5. Oracle Real Applications Clusters (RAC) Constantly improving scalability and HA to protect your data automatically and faster • Oracle Grid Infrastructure • Oracle Clusterware • Oracle ASM • Oracle ACFS • UCP and OCI Connection Load Balancing • Run time load balancing • Database Reliability Framework • Scalable Sequences • Continuous Application Availability • Oracle RAC Sharding • Cluster Health Advisor (CHA) • Service Oriented Buffer Cache Access • Transparent Application Continuity • Commit Cache Messaging • Automatic Hang Resolution • Application Continuity (AC) • Recovery Buddy • LMS CR Server Threads • Node Weighting • Higher Priority for LMS • PDB Service Isolation 11.2 12c 18c • Colocation Tag for Client Routing • Dynamic Services Fallback Option • RAC resource runtime management • Threaded Cache Fusion Processes • Right-Growing Index Performance improvements • In-Memory Transaction Table • Undo Block RDMA Reads 21c • Optimized Broadcast of Commits • PDB resource management • Cache Fusion Hardening • Zero downtime Grid Infrastructure Patching • RDMA for current read blocks • Zero downtime for Planned Outages 19c * Documented features list is selective
- 6. New Generation Oracle RAC Scalability and High Availability New Apps based on Converged DB’s using IoT, Kafka, Blockchain & Traditional packaged Apps Private Network Failover Private Network New Apps Traditional Apps ＆ Copyright © 2021, Oracle and/or its affiliates |
- 7. New Generation Oracle RAC scales IoT workloads New Apps https://www.oracle.com/a/tech/docs/wp-bp-for-iot-with-12c-042017-3679918.pdf IoT Workload Copyright © 2021, Oracle and/or its affiliates |
- 8. PayPal Database Requirements Scalability Performance Availability Oracle RAC • Scale within an RAC Instance • Scale by adding RAC Instances • Rolling Upgrades with Zero Downtime
- 9. 250+ Oracle RAC Clusters Oracle RAC – By the numbers--- 15M+ Execs/Sec 33% Y-o-Y DB Storage Growth 90+ PB Total DB Storage Across Databases 18M+ Logical Reads/Sec (768 Cores, 4 x Oracle X7-8 12 TB memory) 350k User calls/Sec 300k GC/GE messages/Sec High Throughput OLTP RAC Database 2M+ IC packets sent/received < 0.250ms Avg GC message latency < 4ms Avg SQL call latency > 100K Executions/Sec on a table Oracle RAC Deployment at PayPal
- 10. How does Oracle RAC scale ALL kinds of Workloads? Copyright © 2020, Oracle and/or its affiliates 10
- 11. Innovative and Unique : Oracle RAC Cache Fusion Buffer B B B Private Network Private Network B Private Network B Before Cache Fusion With Cache Fusion • Multiple LMSs • Higher Priority • Auto Tune # of LMS • BoC Synchronization • Integration with DRF • Dynamic GRD resizing Oracle 8i Oracle 21c B 2 3 1 2 B 1 Copyright © 2021, Oracle and/or its affiliates |
- 12. Out of the box better Application Performance Right-Growing Index Optimizations • Index split optimizations (12c onwards) • Scalable Sequences (18c) • In-memory Commit Cache (18c) • Delayed ping (19c) Interconnect Optimizations • Light Weight IPC on Generic systems • Improved handling of OS resources like sockets, file descriptors • RDMA on Exadata • Cache Fusion speed improves to <10us, even under load Oracle RAC Cache Fusion Optimizations 3. Data Block Transfer Instance 2 Instance 3 1. Request Instance 1 2. Forward LMS FG LMS 3. Direct Read Instance 2 Instance 3 1. Request Instance 1 FG SGA LMS 2. Grant Copyright © 2021, Oracle and/or its affiliates |
- 13. Utilize RDMA for • Propagating BoC (Broadcast on Commit) • Reduce messages • Reduce CPU usage on LMS • Current read blocks • Very common access pattern in various workloads • Commit Cache messaging • https://www.slideshare.net/AnilNair27/oracle-rac-features-on-exadata (Slide#43) • Results in reduced load on LMS from remote node • Undo block • https://www.slideshare.net/AnilNair27/oracle-rac-features-on-exadata (Slide#44) Cache Fusion Optimizations on Exadata Copyright © 2021, Oracle and/or its affiliates |
- 14. Efficient Operating System Resource Management Adaptive Oracle Database Resource Management Efficient and Adaptive Resource Management CPU Network Disks RAM Global Resource Directory Copyright © 2021, Oracle and/or its affiliates |
- 15. Adapt to Modern CPU Architectures DRAM Process Process Core Core Core Core Process Process L2 Cache Process Process Process Process Thread Thread Thread Thread Thread Thread Thread Thread L1 Cache L1 Cache L1 Cache L1 Cache Copyright © 2021, Oracle and/or its affiliates | Modern CPUs have multiple cores designed to run multiple processing threads at the same time
- 16. Leaner Threaded Processes benefit from current CPU architectures • Threaded LMS process dynamically started based on need • Process runs in higher Priority (RR) while Threads run in Normal Priority TS • Monitored and changed via DRF (Database Reliability Framework) • Continuously adapts based on incoming workload Oracle RAC Optimized Priority for Critical Processes $ps -eLo 'pid tid cls cmd ' |grep '_lm' |grep sales |sort -k4 6970 6970 TS ora_lmon_rws17c 6972 6972 TS ora_lmd0_rws17c 6974 6977 RR ora_lms0_rws17c 6979 6979 RR ora_lmhb_rws1 6974 6981 TS ora_lms1_rws17c 6974 6983 TS ora_lms2_rws17c LMS runs in RR LMS threads run in TS Copyright © 2021, Oracle and/or its affiliates |
- 17. More fine-grained process threads introduced over the years • Dia0 – Hang Manager • Gcr0- Global Conflict Resolution thread • Insv – DG broker communications • IPC0- Basic messaging • Ping – Interconnect measurement • ACMS – CF Writes Oracle RAC Fine Grained Process Model $ps -ef |grep 'sales' |egrep "gcr|lmd" |sort -k8 racusr 15871 1 0 Jun25 ? 01:40:56 ora_gcr0_sales2 racusr 16925 1 0 Jun25 ? 00:26:18 ora_gcr0_sales2 racusr 16931 1 0 Jun25 ? 00:25:47 ora_gcr1_sales2 racusr 10828 1 0 Jun25 ? 00:26:13 ora_gcr2_sales2 racusr 11587 1 0 Jun25 ? 00:25:59 ora_gcr3_sales2 racusr 15593 1 0 Jun25 ? 00:41:17 ora_lmd0_sales2 racusr 15603 1 0 Jun25 ? 00:32:55 ora_lmd1_sales2 Copyright © 2021, Oracle and/or its affiliates | LMD threads GCR threads
- 18. Significantly contributing to Oracle RAC 19c Availability 18c 19c 11g 12c 4x faster 7x faster Oracle RAC reconfiguration times reduced by • 4x between Oracle RAC 11g and 12c • 7x between Oracle RAC 11g and 19c Selection of contributing features: • Remastering servers (12c) • Recovery Buddy (12c) • Support for 100 LMS’s (12c) • LMS’s run as OS threads (18c) • More Reconfig Optimizations (19c) Copyright © 2021, Oracle and/or its affiliates |
- 19. ORACLE RAC – Zero Downtime Out of Box <10 sec Optimized DRM Zero Downtime with RAC Rolling Patches Reconfiguration Brownout times
- 20. ORACLE RAC - Reconfiguration Customizations • Alter system set db_cache_size = • Reconfiguration of < 5 sec • 1TB cache shrink ~ 20 min 50% improvement
- 21. Optimize usage of OS Resources. – Memory, Huge Pages Available system pagesizes: 4K, 2048K Supported system pagesize(s): PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s) 2048K 128000 66562 66562 NONE Initial number of CPU is 72 Number of processor cores in the system is 36 Number of processor sockets in the system is 2 NUMA system with 2 nodes detected Copyright © 2021, Oracle and/or its affiliates |
- 22. Optimize usage of OS Resources. -- Network Cluster Communication is configured to use IPs from: GPnP IP: 169.254.12.142 Subnet: 169.254.0.0 KSIPC Loopback IP addresses(OSD): 127.0.0.1 KSIPC Available Transports for cluster communication: UDP:TCP KSIPC: Client: KCL Transport: UDP KSIPC: Client: DLM Transport: UDP KSIPC CAPABILITIES :MGA:IPCLW:GRPAM:TOPO:DLL:STATSFW:CRTRK KSXP: ksxpsg_ipclwtrans: 2 UDP cluster interconnect IPC version: [IPCLW over UDP(mode 3) ] IPC Vendor 1 proto 2 Copyright © 2021, Oracle and/or its affiliates |
- 23. Database Reliability Framework – Adapt to changing workloads Dynamic DRM Consider Block congestion statistic for Cache Fusion Copyright © 2021, Oracle and/or its affiliates |
- 24. Near Zero Reconfiguration time with Recovery Buddies Cache Fusion provides complete isolation between PDBs Continuous improvements in every release Oracle RAC High Availability Optimizations Smart Fencing ensures majority of work survives Hang Manager automatically detects and resolves hangs and deadlocks Copyright © 2021, Oracle and/or its affiliates |
- 25. Requires no changes to the Application • Incoming connection requests are load balanced across all instances • Public network and instance health is constantly monitored by Grid Infrastructure • On failure, GI will relocate the VIP • Existing connection requests will receive error as GI will send a GARP • ONS consumers like SCAN listeners, Connection Pools get notified of failure Database Clients – Load Balancing and Failure Handling ODP.Net, OCI, Connection Pool Single Client Access Network New Connection request Scan Listener routes to DB listener on first node Instance or Public Network failure GI relocates VIP to another instance Connection attempted on VIP Instant Error instead of waiting on TCP timeout Retry Connection ONS Event Notification Gratuitous ARP Arp Table update Copyright © 2020, Oracle and/or its affiliates |
- 26. Oracle RAC Deployment Pattern at PayPal USERs MICRO SERVICES • REDO 300 MBytesPs • Interconnect 200 MBytesPs • Write ~ 50K IOPs/ 5 GBPs • Read ~ 50K IOPs/ 5 GBPs • Execs/s ~100k per instance • Gets/Block changes ~ 5m Read Write Service Read only Service Sync Sync Sync
- 27. Oracle RAC Scalability Optimizations Smart Cache Fusion monitors and dynamically chooses optimal path SSD Service Oriented Buffer Cache Cache Fusion monitors and maintains Services and Buffer Cache affinity Pluggable Database Service optimization Optimized Singleton Workload Scaling provides consistent performance after planned service failover Copyright © 2021, Oracle and/or its affiliates | Continuous improvements in every release
- 28. 15M+ Execs/Sec Extreme Availability and Performance Oracle RAC - Performance Characteristics < 0.250ms Avg GC message latency 300k GC/GE messages/Sec < 4ms Avg SQL call latency
- 29. • Older versions leverages DBMS_CACHEUTIL to grab affinity • Reduce unplanned performance impact with traffic shifts • Instance stop/start reconfiguration impact • Mileage will vary ORACLE RAC – Performance Customizations B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A Global Resource Directory B Buffer M Admin B A
- 30. Summary PayPal workloads scale efficiently with Oracle RAC
- 31. Preview of what’s coming up in Oracle RAC 21c Copyright © 2020, Oracle and/or its affiliates 31
- 32. Zero Downtime Grid Infrastructure Patching Zero-Downtime Oracle Grid Infrastructure (GI) Patching • Patch Oracle GI without interrupting database operations • Patches are applied in a rolling fashion (one node at a time) while the database instance(s) on that node remain up and running • Limited availability with version 19.7 and higher • General availability with Oracle 21c + RU & RUR support https://docs.oracle.com/en/database/oracle/oracle- database/19/cwlin/about-zero-downtime-gi-patching.html#GUID- 253CF841-DAAD-4B13-A1F2-6F83EF3FDE47 Copyright © 2021, Oracle and/or its affiliates |
- 33. Oracle RAC Cache Fusion Hardening • This feature attempts to recover from discrepancies received from the sending instance • In the past, any such discrepancy between Cache Fusion processes may cause instance to crash with (ORA-481) • Cache Fusion will try and attempt to fix the discrepancy by retrying the buffer request from the sending instance Copyright © 2021, Oracle and/or its affiliates |
- 34. Zero Downtime for Planned outages During planned shutdown • Distribute Resource Admins before Instance shutdown • Distribution before shutdown does not require any recovery on the surviving instances • Effectively reduces the time spent on reconfiguration during planned outages to zero M M M M M M M M Copyright © 2021, Oracle and/or its affiliates | A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A Admin
- 35. Both traditional packaged Applications along with new apps based on IoT, Kafka, Blockchain Benefits of Deploying on Oracle RAC Highly Available Scale within a Database Instance Scale by adding Instances Requires No Changes to Application Client Application Failover Connection Load Balancing Scale In-Memory (Analytics) Scale Multitenant (Microservices) Complete Isolation in Converged Envs Security New Generation Oracle RAC provides HA and Seamlessly Scales Copyright © 2021, Oracle and/or its affiliates |
- 36. Thank you Oracle RAC Product Development Team & PayPal Engineering Team Copyright © 2021, Oracle and/or its affiliates |
- 37. Questions?