![29ni.com
AdministeringApplication Nodes
Example:
[Instance Specific]
s_temp=/opt/apps/r12/d3ebiz/apps/inst/fs2/inst/apps/d3ebiz_d3ebizui1/temp
s_contextname=d3ebiz_d3ebizui1
s_hostname=d3ebizui1
s_domainname=natinst.com
s_cphost=d3ebizcm1
s_webhost=d3ebizui1
s_config_home=/opt/apps/r12/d3ebiz/apps/inst/fs2/inst/apps/d3ebiz_d3ebizui1
s_inst_base=/opt/apps/r12/d3ebiz/apps/inst
s_display=d3ebizui1:0.0
s_forms-c4ws_display=d3ebizui1:0.0
s_ohs_instance=EBS_web_d3ebiz_OHS2
s_webport=8000
s_http_listen_parameter=8000
s_https_listen_parameter=4443](https://image.slidesharecdn.com/a3600d8c-e3ef-4b4b-a079-0c8060d648ff-161113040134/85/AOUG_11Nov2016_Challenges_with_EBS12_2-29-320.jpg)
![30ni.com
AdministeringApplication Nodes
Example:
[Services]
s_web_applications_status=enabled
s_web_entry_status=enabled
s_apcstatus=enabled
s_root_status=enabled
s_batch_status=enabled
s_other_service_group_status=disabled
s_adminserverstatus=disabled
s_web_admin_status=disabled](https://image.slidesharecdn.com/a3600d8c-e3ef-4b4b-a079-0c8060d648ff-161113040134/85/AOUG_11Nov2016_Challenges_with_EBS12_2-30-320.jpg)
Ad
More Related Content
What's hot (19)
Viewers also liked (12)
Ad
Similar to AOUG_11Nov2016_Challenges_with_EBS12_2 (20)
Ad
AOUG_11Nov2016_Challenges_with_EBS12_2
- 1. ni.com Online Patching Challenges with EBS 12.2.5 Sean Braymen & Miklos Lorant
- 2. 2ni.com About National Instruments Founded: 1976 Corporate Headquarters: Austin, Texas Industry: Test and Measurement Our Mission: We equip engineers and scientists with systems that accelerate productivity, innovation, and discovery. Revenue: $1.23 billion in 2015 Global Operations: Approximately 7,100 employees; operations in almost 50 countries IT Development Centers: • Austin, Texas • Debrecen, Hungary • Penang, Malaysia • San José, Costa Rica • Shanghai, China
- 3. 3ni.com Agenda • Review of 12.2 Architecture and ADOP Procedures • 12.2 Architectural Changes and Philosophy • Online Patching o File System Editioning o Edition Based Redefinition o ADOP Patching Cycle • Custom Development Standards • Full vs Runtime Compliance • Custom Development Options • Lessons Learned • Database Object Grants • DB_Domain • Executing Autoconfig • Administering Application Nodes • Challenges with Online Patching • ADOP - Useful Utilities
- 4. ni.com
- 5. 5ni.com Copy the Code, NOT the Data • EBS uses both the file system and the database to store the code and data that make up the application • Code o Stored both on the File system & In the Database o Any code object changed in a patch is copied • Data o Stored both on the File system & In the Database o Application data is NOT copied by a patch
- 6. 6ni.com Downtime Redefined as “Cutover” • Cutover is the time taken to switch users from the production system to the newly patched copy • Cutover changes the unit of measure for downtime • Measured in minutes NOT hours or days • Cutover time is very predictable • The time taken to bounce the Middle Tiers Changing the Unit of Measure for Downtime
- 7. 7ni.com Downtime Limited to Short Cutover • Users are logged off the production Instance • When users reconnect they are directed to the newly patched system E-Business Suite 12.2.3 Patches •12.2.4 •12.2.5 •…. E-Business Suite 12.2.3 Production Patched Production
- 8. 8ni.com • Downtime Patches • No online users • Wall clock time very important • Consumes all resources • Upgrade designed to run as fast as possible • Online Patches • User remain online • Wall clock time is no longer an overriding concern • Online users share resources • Data upgrades designed to not affect the running application Changing the Patching Paradigm
- 9. 9ni.com Online Patching - High level Architecture
- 10. 10ni.com 12.2 Architecture Dual File System and Edition-Based Redefinition Synchronization Managed Automatically Edition-Based Redefinition Non-Editioned File System Developer 10.1.2 COMMON_TOP APPL_TOP INST_TOP Oracle HTTP Server (OHS) WebLogic Server (WLS) Run File System Developer 10.1.2 COMMON_TOP APPL_TOP INST_TOP Oracle HTTP Server (OHS) WebLogic Server (WLS) Patch File System PATCH_TOP APPL_TOP_NE LOGS
- 12. 12ni.com • Run file system • Used by online users • Stores a complete copy of all Applications and Middle Tier code • Logically mapped to either fs1 or fs2 • Patch file system • Used by patching tools • Stores a complete copy of all Applications and Middle Tier code • Logically mapped to either fs1 or fs2 • Non-Editioned file system • Used for data files e.g.: data import/export files, log files, report output files • Only stores data files Online Patching uses a Dual File System fs1 and fs2 switch Run and Patch designation during the cutover phase of an Online Patching cycle fs1 Run fs1 Cutover fs1fs2 PatchPatch fs2 Run
- 14. 14ni.com 11gR2 Edition-Based Redefinition (EBR) • Key enabling technology for Online Patching • Enables the online upgrade of the application definition • Allows an application to efficiently store multiple copies of its code and schema in the same database • Provides an isolation mechanism that allows pre-upgrade and post-upgrade schemas to co-exist • Changes to database objects are made in the isolation of an “Edition” • Changes to database objects do not affect the running application
- 15. 15ni.com Online Patching is Enabled by 11gR2 Edition-Based Redefinition Multiple Copies of Database Code Objects Can Coexist • Client code connects to an “edition” of the database • Run Edition o Used by online users o Never changed by a patch • Patch Edition o Used by patching tools o Changes do not affect the running application • Patch Edition becomes the Run Edition at Cutover 15 DBA Business Users
- 17. 17ni.com Online Patching Cycle - Overview Understanding the Online Patching Cycle • The Basics • Remove obsolete objects Cleanup • Restart application on Patch Edition Cutover • Compile invalid Objects • Wait for a good downtime window Finalize • Apply one or more patches to the Patch Edition Apply • Copy the production application code • Create a new Patch Edition in the database Prepare Users Online Users OnlineUsers Offline • Online Patching is used to apply all patches in 12.2 • Online Patching cycle includes 5 major phases • Application is only offline during the Cutover phase
- 18. 18ni.com Custom Development with Online Patching
- 19. 19ni.com Full Compliance vs Runtime Compliance • Full Compliance • Development meets all standards to support runtime and online patching operations • Specialized deploy methodology to implement changes in the patch edition/file system o See Doc ID 1577661.1 for all online patching compliance standards • All Oracle eBusiness Suite products meet full compliance • Runtime Compliance • A subset of the full compliance standards that allow a product to operate correctly in R12.2 • Customizations and third-party tools may only meet runtime compliance
- 20. 20ni.com Pro/Cons of Online Patching Full Compliance • Oracle’s Deployment Methodology • Doc ID 1577661.1 – Section 1.4 • Standard application development is executed directly against the run edition • Option #1: Apply Customizations as Online Patch. When application development is completed, a custom patch is created and deployed in an ADOP cycle • Option #2: Apply customizations as a downtime patch. Custom code is added to run edition in a downtime window • Advantages • Higher guarantee of system stability • Disadvantages: • Changes to the Applications are less agile – Both paths require downtime • Option #1 requires the creation of patches. Steps vary based on AD/TXK patchset level and database objects being modified. See Part 2 of Doc ID 1577661.1
- 21. 21ni.com Disadvantage of Runtime Compliance • Deployment Methodology • Deploy code directly to the run edition • Advantages • Higher degree of flexibility. Code changes can be done without system wide downtime • Removes the learning curve for developers • Disadvantages • Extra maintenance required before ADOP cycle. Must execute a process to synchronize run and patch file systems o Delta 7 and below: fs_clone or custom process o Delta 8: Possibly taken care of in the prepare phase • Higher risk of system stability issues during code deploys
- 23. 23ni.com Database Object Grants • Doc ID 1987947.1 • Symptoms • Grants cannot be performed in the run edition when the application is being used • Granting privileges on an object may cause object invalidations in the current edition • Problem Description • When grants are given to "Stub objects" it invalidates all dependent stub objects. • “Stub objects" are simply pointers to an actual object definition from an ancestor edition that is still being inherited by the given edition.
- 24. 24ni.com Database Object Grants Solutions • Option #1: Execute an ADOP cycle when granting privileges on the APPS schema to other users • Option #2: Leverage new API for granting privileges with minimum invalidations • API: AD_ZD.GRANT_PRIVS
- 25. 25ni.com Database Object Grants • Syntax: ad_zd.grant_privs (X_PERMISSIONS in varchar2, X_OBJECT_NAME in varchar2, X_GRANTEE in varchar2, X_OPTIONS in varchar2 default NULL); • X_PERMISSIONS: comma-seperated permission list • X_OBJECT_NAME: name of APPS object or APPS synonym • X_GRANTEE: grantee schema or role • X_OPTIONS: grant options, example: 'WITH GRANT OPTION'
- 26. 26ni.com DB_Domain • Specifies the logical location of the database within the network structure • Rapid Clone and ADOP are making assumptions about the DB_Domain • Assumption #1: DB_Name + DB_Domain is less than 8 characters o If this condition is not met, Rapid clone on the Application tier will not execute o Workaround: Unset DB_Domain, run rapid clone, reset DB_Domain, run Autoconfig • Assumption #2: The DB_Domain has the same value as the host domain o In our case: – DB_Domain = WORLD – Host Domain = natinst.com o ADOP builds a JDBC connect string during the ADOP patching cycle o Workaround: Create a service for db_name + host_domain – d1ebiz.natinst.com
- 27. 27ni.com ExecutingAutoconfig • Application services should be down while running autoconfig • Autoconfig has to be executed on ALL application tier nodes • Autoconfig cannot be executed in parallel on multiple application tier nodes • While the autoconfig script will show a successful completion, data is not properly inserted into FND_OAM_CONTEXT_FILES • This will lead to potential issues during the next ADOP cycle in which nodes may need to be abandoned
- 28. 28ni.com AdministeringApplication Nodes • Adding Application Nodes • Doc ID 1375769.1 • Application nodes cannot be added or removed when application services are running • Major steps • Setup ssh between Primary APPS Tier and New Node • Run adpreclone.pl appsTier for both patch and run file system • Create Oracle Inventory • Prepare Pair File for adclonectx
- 31. 31ni.com AdministeringApplication Nodes Execute the Add Node Utility /opt/apps/r12/<db_name>/apps/<run_fs>/FMW_Home/webtier/p erl/bin/perl ./adclonectx.pl addnode contextfile=<full path of the cm1 node's Run File System Context file> pairsfile=<full path of custom pairsfile> dualfs=yes
- 32. 32ni.com AdministeringApplication Nodes • Changing apps.conf and mod_wl_ohs.conf perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl -contextfile=<full path of the <host> Run File System Context file> -configoption=addMS -oacore=<new host>.natinst.com:<oacore port number> -oafm=<new host>.natinst.com:<oafm port number> -forms=<new host>.natinst.com:<forms port number> -formsc4ws=<new host>.natinst.com:<formsc4ws port number>
- 33. 33ni.com AdministeringApplication Nodes • Change context file entries (if needed), eg: Change following entries from enabled to disabled on CM nodes s_oacorestatus s_formsstatus s_oafmstatus • Run autoconfig
- 34. 34ni.com AdministeringApplication Nodes • Adding Managed Servers • Doc ID 1905593.1 • New managed servers can be added when the applications are running • However, they will not be used until the applications are stopped and autoconfig is executed perl $AD_TOP/patch/115/bin/adProvisionEBS.pl ebs-create-managedserver -contextfile=$CONTEXT_FILE -managedsrvname=forms_server17 -servicetype=forms -managedsrvport=7412 -logfile=/tmp/addforms_server.log
- 35. 35ni.com AdministeringApplication • E-Business Suite Technology Codelevel Checker (ETCC) • Rapid Install StartCD 50 for EBS 12.2 comes with a 11.2.0.3 database • Since Feb. 22, 2016, Rapid Install StartCD 51 is available. It comes with 12.1.0.2 DB. You can download as Patch 22066363 • Since Sept 15, 2016, 12.2.6 is available. New Installation and Upgrade (upgrading to Oracle E-Business Suite Release 12.2.6 from Release 11i, 12.0, or 12.1) customers should use StartCD 12.2.0.51. (Doc ID 2114016.1) • Weblogic 11.1.1.7 upgrade to 11.1.1.9 was not possible till we applied all ETCC recommended patches
- 36. 36ni.com Challenges with Online Patching • Serial number in context file should match the value of database copy. If not, adop can fail (Doc ID 1916658.1) • adop phase=fs_clone • In our environment, performance is at least 3 hours • The addition of more application nodes increases the time for the procedure to complete • Recommended to be executed after any configuration changes • Not all files get transferred on slave nodes o oaea_wls.properties • adop fs_clone allnode=yes|no • adop fs_clone force=yes
- 37. 37ni.com Challenges with Online Patching • How it supposed to look like (eg.:): SSO_SERVER_RELEASE=11 SSO_SERVER_URL=https://winnie-app.natinst.com:443 SSO_SERVER_TYPE=OAM LOG_CONFIG_FILE=DEFAULT APPL_SERVER_ID=3DA0E4FD00B9625FE05400144FFB48B032374020223902728214531018323218 OAM_LOGOUT_URL=/accessgate/logout CONNECTION_REF=jdbc/OAEADatasourceDS WEBGATE_LOGOUT= oracle.apps.fnd.sso.WebEntries=DEFAULT • How it was looking like (everywhere, except master node): SSO_SERVER_RELEASE= SSO_SERVER_URL= SSO_SERVER_TYPE= LOG_CONFIG_FILE=DEFAULT APPL_SERVER_ID= OAM_LOGOUT_URL= CONNECTION_REF= WEBGATE_LOGOUT= oracle.apps.fnd.sso.WebEntries=DEFAULT
- 38. 38ni.com Online Patching – Useful Utilities • adopmon - utility to monitor the progress of adop session. • adopreports - Online Patching Diagnostic Reports utility, $AD_TOP/bin/adopreports, can be used to help diagnose issues or simply gather information about the status of your system. • txkADOPValidations.pl - before prepare, cutover or fs_clone, this tool can validate the instance if it is ready for those phases perl $AD_TOP/patch/115/bin/txkADOPValidations.pl -contextfile=$RUN_CONTEXT_FILE - patchctxfile=$PATCH_CONTEXT_FILE -phase=fs_clone -logloc=/tmp • adopscanlog - utility makes the work easier and analyzes adop log directories for errors and warnings
Editor's Notes
- #18: Online patching has 5 cycles: First, the system is prepared by creating a copy of production code and creating a new edition in the database Then, you apply one or more patches to the patch edition with adop Next, the finalize phase is executed which includes compiling invalids Then you wait for a time to cutover from the run filesystem and edition to the updated patch filesystem and edition Cutover – is in red as this is the only time the system is down You execute cutover, when it is convenient for your environment Finally you can execute the cleanup phase any time after cutover