Always test the patch on a test system prior to applying to the production database. Because of the rare possibility of problems being encountered, perform a “cold” system backup of your test instance, if one doesn’t already exist.
Patch types Oracle produces
Maintenance Pack: A Maintenance pack is a consolidation, or bundling, of patches for an entire product area. Maintenance packs introduce new functionality in a given product and are particularly useful for customers who are beginning their implementation. They can start with the application of a Maintenance Pack, such as the 11.5.3 CD Pack versus applying the 11.5.1 CD Pack followed by the individual patches released since 11.5.1’s initial availability. Maintenance Packs are cumulative, so a customer does not need to apply 11.5.1, 11.5.2 and 11.5.3 to get to the latest code, they can start with 11.5.3 (or the latest version available). Maintenance Packs can be downloaded from MetaLink but are quite large and should be ordered as CD Pack from the Oracle Store. A Maintenance Pack is the largest or top most level of patching done in Oracle Applications. This type of patch contains all of the product specific family packs for the specified release version.
The readme file for a Maintenance Pack contains the list of patch’s that are included in the Maintenance Pack and can be downloaded from MetaLink.
The product minor version (11.5.1, 11.5.2, 11.5.3, etc) can change by the application of Maintenance packs.
Special considerations for upgrade customers:
Maintenance packs are not for upgrading from previous version of the Applications, such as 10.7 or 11.0.x. In these cases, customers must use RapidWiz and AutoUpgrade utilities and complete all the steps specified in the Upgrade Manual, before applying the Maintenance pack. Customers must obtain the CD Pack from the Oracle Store because it contains the RapidWiz. The RapidWiz cannot be downloaded from MetaLink
Family Pack: A Family Pack patch can introduce new functionality in a given product area. It also is a consolidated set of patches specific to a product family area, such as AOL or HRMS. Family Packs are cumulative, meaning all product family packs need to be applied in sequence. Family Packs are denoted with a letter, such as A, B, C etc., “A” preceding “B” and “B” preceding “C” and so on. The functionality changes included in the Family Packs are included in subsequent Maintenance Packs.
Mini-Pack: The terms megapatch, patchset, and mini-pack are interchangeable terms and refer to a group of bug fixes that are applied at one time. Mini-packs can be released for a single product or a group of related products (e.g. Purchasing mini-pack, Foundation mini-pack, Accounts Payable mini-pack, etc.). They are generally given alphabetical suffixes that provide their position in the sequence of a product’s lifecycle such as AP.A, AP.B, AP.C, etc. Mini-packs are cumulative and can contain new functionality.
One-off Patch: A "one-off" patch addresses a single fix or enhancement. One-off patches can introduce new functionality and is noted in the readme file for the patch. One-off patches are usually issued when an issue requires an immediate code change that can’t wait for distribution in any other form.
All ERP patches are delivered with the software modules that are needed, including the dependencies for that module. Some modules, such as forms, have a large number of dependencies. Even a small change like correcting a LOV for a form can result in the one-off patch to include dozens of other modules to satisfy the dependencies.
Diagnostic Patch: A diagnostic patch is sent to a customer to assist Oracle Support Services and Oracle Development obtain diagnostic information. The patch is provided when a product failure cannot be reproduced in an OSS environment and customer specific data is required to continue the investigation.
Interoperability Patch: Interoperability patches can be required when an Oracle Applications instance is migrated to a newer version of the technology stack. The patch is typically included in the DB and Forms technology stack upgrade. An example of this was when Oracle Applications Release 11 was initially released, it required version 8.0 of the DB. Later on, when the Applications were certified with 8i, the Applications required an interoperability patch to work with 8i. Interoperability patches are quite common in Oracle Applications releases 10.7NCA, 11.0.X and 11i due to the to synchronization Oracle Applications modules with the core Oracle database and tools. This type of patch can be created when an upgrade occurs in the Oracle software stack and Oracle Applications require additional modules to maintain functionality.
Translated Patch: A fully language translated patch. From R11i on, all applications patches are automatically translated into 28 different languages. The timing and availability of these patches may vary.
A class of patches that contain legislative data has an additional driver called hrglobal, which may need to be applied. Also, for some groups of patches, it may be beneficial to merge the patches into one set of driver files. Depending upon your implementation, you may also need to deal with multi-language patches and multi-node patching. These topics are discussed in the following sections.
Applying Legislative Patches
For Oracle Payroll customers, there is another category of patch required by the system. The hrglobal patch supports the legislative requirements of multiple countries. Given the nature of this patch, it is updated frequently by Oracle. It is often a post-patch requirement for the mandatory patches released for Oracle Payroll.
To find the latest copy of the hrglobal patch, view MetaLink Note 145837.1. This note will contain the latest patch number for the hrglobal patch, along with a link to the patch installation instructions and a change history for the patch. The hrglobal patch can be downloaded from MetaLink like any other patch. Review the patch’s readme file for required prerequisites.
After unpacking the patch, the adpatch utility can be run to install the patch’s u driver. In addition to the u driver, these patches contain a special hrglobal driver. As a result of these differences, there are additional considerations for applying this type of patch.
Once the u driver has been applied, the DataInstall Java utility needs to be run in order to select needed legislations for install. The syntax for this command is as follows:
jre oracle.apps.per.DataInstall apps apps_password thin
[hostname]:[dbport]:[oracle_sid]
When the DataInstall utility has been executed, the Applications DBA will need to select all relevant legislations. Figure 5-6 shows the main menu for DataInstall.
DataInstall Main Menu
1. Select legislative data to install/upgrade
2. Select college data to install/upgrade
3. Select JIT/Geocode or OTL to install/upgrade
4. Exit to confirmation menu
Figure 5-6. The DataInstall Main Menu
Select option 1 to choose which legislative data to install or upgrade. From the resulting menu, shown in Figure 5-7, you should choose to install any legislative data marked as installed. Note that the selection numbers will change depending upon your version of the hrglobal patch. Check the numbers for the appropriate data.
# Localisation Product(s) Leg. Data? Action
1 Global Human Resources Installed
2 Australia Human Resources
3 Australia Payroll...
55 United States Human Resources Installed
56 United States Payroll Installed
Figure 5-7. The DataInstall legislative data submenu
Select the legislative data to be installed by entering the localization number and I. If an incorrect number is selected, you can correct the mistake by entering that number with a C to clear the action.
After all legislative data is marked for install, return to the main menu to select any required college data. When all college data is selected, return to the main menu and select 4 to exit the utility. Upon exiting, an Actions Summary will be displayed. Review that summary to ensure that all required actions have been selected.
The final stage of the legislative patch is to run the adpatch utility to apply the hrglobal driver. This driver is copied to the $PER_TOP/patch/115/driver directory by the patch’s u driver. The same adpatch options for applying other drivers should be used for the hrglobal driver.
Using AD Merge
When applying a group of large patches, such as a Maintenance Pack and a cumulative update, some performance benefits can be incurred by using the AD Merge utility to combine the patches into one patch. From my personal experiences I would say merging the best time reducing feature. People most of the times keep asking how do we know which pathces to merge and which not. Well my answer always would be, depends on Analysis.which is done with the readme and the docs associated with the patch. And a thorough analysis in the earlier stage will save you the downtime and make the patching smoother and better.
The set of patches to be merged should be copied to a common directory. After the patches are unbundled, the AD Merge utility can be run against the patches. Here is an example:
admrgpch /source_dir /target_dir
The completed merged driver files found in the target directory can be applied as a standard patch would be applied. The merged driver files will have a name like u_merged.drv. A log file, admrgpch.log, will be created in the directory where the utility was run.
For more information, see MetaLink Note 228779.1, “How to Merge Patches Using admrgpch.” The admrgpch utility can be run with several parameters, shown in Table 5-3.
Option Purpose
s Specifies the source directory containing compressed patch files.
d Specifies the destination directory for merged patch files.
verbose Controls the level of detail included in admrgpch output.
manifest Specifies a text file containing the list of patch files to be merged. This is useful if the source directory includes a large number of patch files.
logfile Specifies the log file to contain the output from admrgpch utility.
merge_name Specifies the name of the merged file. This defaults to “merged”, and it should be changed to be more descriptive.
Table 5-3. admrgpch Options
When using this utility, thoroughly test the resulting patch.
Applying NLS Patches
For E-Business Suite installations with multiple language requirements, there are patches available for each additional language. Each required NLS patch needs to be applied to Oracle Applications. Oracle provides some recommendations for dealing with NLS patches; these are outlined in MetaLink Note 174436.1.
The U.S. version of the patch should be applied before any of the translation patches. The translation patches may be applied without downtime to the entire system if users of the affected language are not active.
Using admrgpch, it is possible to merge all U.S. patches into one patch, and then merge all non-U.S. patches into a separate patch. Depending upon the application configuration, some variation of this approach may be necessary.
Performing Multi-Node Patching
There are a couple of options available to optimize patching for multi-node environments. As of Oracle Applications 11.5.10, the system can be designed with a shared application-tier filesystem. The shared application filesystem contains the application’s APPL_TOP, COMMON_TOP, and ORACLE_HOME. (MetaLink Note 233428.1 describes sharing the application-tier filesystem.) As a result of this configuration, patching the shared filesystem applies the patch to all nodes.
Prior to 11.5.10, a shared APPL_TOP did not include the ORACLE_HOME. For these systems, Forms and iAS patches must be applied to each Form and Web Node.
In order to increase the performance of the patching process, Distributed AD will execute workers on remote nodes in a multi-node implementation. Distributed AD improves scalability and resource utilization. Distributed AD is only available with AD Minipack H or later, and with a shared Application Tier Filesystem or shared APPL_TOP. More information on this feature can be found in MetaLink Note 236469.1.
If a shared Application Tier Filesystem is not in use, each filesystem will need to be patched separately. A patched filesystem can be cloned to another node if the downtime required to patch the node exceeds the time required to clone the filesystem.
Patch drivers have different requirements when applying them in a multi-node environment. The c driver must be applied to all APPL_TOPs, the d driver is applied on the Admin Node, the g driver is applied to all APPL_TOPs unless the APPL_TOP is only the Admin Node, and the u driver is applied to all APPL_TOPs on all nodes.
Monitoring and Resolving Patching Problems
Patching problems manifest themselves in many different ways. Typically the adpatch session will display an error or will appear to be hung on one task for a long period of time. The first step in resolving the issue is to review the adpatch log file and associated worker log file. Next, the reason the worker failed must be determined and resolved. After resolution has been obtained, adctrl can be used to continue the patching process.
Reviewing Log Files
During and after the application of patches, it is helpful to review log files of the adpatch session and its workers. These files are found in the $APPL_TOP/admin/$CONTEXT_NAME/log directory. The adpatch log filename is specified during the patch process. See the “Using AD Patch” section earlier in the chapter for more details.
In order to monitor the patch from a telnet session other than the one where the patch was started, a simple UNIX command such as tail -f u[patch#].log will display information as it is written to the log file. This is a useful means for monitoring the progress of a patch that is being applied.
The log files for the workers will be named adwork[xxx].log, where [xxx] is the number of the patch worker process. If a particular worker has failed, examine the related log file for detailed information. This information can be researched on MetaLink or used to open an SR with Oracle Support.
For example, the log file listing for the u driver of patch 11112, applied through adpatch using 5 workers, may look like this:
$ls
adwork001.log
adwork002.log
adwork003.log
adwork004.log
adwork005.log
u111112.log
Using AD Control
The administrative tool used to manage patch workers is AD Control, or adctrl. Frequently workers will fail or hang, which will require the Oracle Applications DBA to interface with adctrl. (Common patching errors will be covered later in this chapter.)
AD Control menu options will vary depending upon the AD patch version applied to the instance. When logged in as the application owner on the Admin Node, execute adctrl to display the menu options shown in Figure 5-8.
AD Controller Menu
----------------------------------------------------
1. Show worker status
2. Tell worker to restart a failed job
3. Tell worker to quit
4. Tell manager that a worker failed its job
5. Tell manager that a worker acknowledges quit
6. Restart a worker on the current machine
7. Exit
Figure 5-8. AD Controller Menu
To execute an adctrl menu option, simply type the menu option and press Enter. If options 2–6 are chosen, either specify the number of the worker that requires action, or press Enter for the action to be executed for all workers.
The “Skip Worker” menu option is a hidden adctrl menu option. If a worker needs to be skipped, start adctrl, enter 8, and then enter the worker number. Only use this option if advised by Oracle Support.even if we use this with out any support advise , be sure what the worker was going to do and how for the patch has gone .As some times you may skip a worker but that would have been doing some work which is required in the further process of the patch.Hence we may go into deeper trouble skipping the workers just to continue the patch.
Resolving AD Patch Worker Failure
If a worker has failed, the adpatch session will normally display a failedworker message. The status of the worker may also be determined using adctrl. If a worker has failed, the worker error can be obtained by viewing the worker log file. Once the worker issue has been resolved, use adctrl to restart the worker.
If a worker has failed, and it is determined that the step the worker was trying to execute may be skipped, the hidden option 8 of the adctrl menu, “Skip Worker,” may be used to skip the worker. It is only advisable to do this if the step is not critical to the environment being patched.
The following are common worker failures that will be seen by the Applications DBA during patching. The error messages will be displayed by the adpatch session or in the worker log file:
Error message: ORA-01013: user requested cancel of current operation
Resolution to error: If this error occurs, simply use adctrl to restart the worker on the current machine.
Error message: Patch not applied successfully, adpatch did not cleanup its restart files (*rf9).
Resolution to error: If this error occurs, execute the following as the instance owner:
$cd $APPL_TOP/admin/$CONTEXT_NAME
$mv restart restart_old
$mkdir restart
After cleaning up the restart files, you may then restart the adpatch session using adpatch.
Error message: ERROR: Javakey subcommand exited with status 1
Resolution to error: If this error occurs, the identity.obj file needs to be re-created.Recreate the identity.obj file. Then, use adctrl to restart the failed worker.
Error message: No error message is displayed; rather the worker log file states that the worker is complete, yet adctrl indicates that the worker is still running.
Resolution to error: This patching problem occurs when the worker is complete, but did not update patching tables correctly to notify the adpatch session that it has finished. In this scenario, the adpatch session is still waiting for the finish return code from the worker. When this occurs, use adctrl to fail the worker, then restart the worker.
Any form, library, or report that fails to generate during the patch process can be regenerated manually after all patching and post-patching steps have completed. If the object still fails to compile, open an SR.
Additional Tips for Resolving Patching Issues
If a patch has hung or workers have failed, and the reason for this failure cannot be determined, it is advisable to check the number of invalid objects in the database. If the number of invalid objects is high, recompile the invalid objects in parallel and restart the patching session.
If the adpatch session is hung, and all other methods for resolution have been executed, it may be necessary to bounce the database and restart the patch session. This method for resolving patching issues is sometimes necessary, especially when applying large patches, such as Maintenance Packs.
If a failure occurs during the application of a patch, it may be necessary to apply another patch to resolve the issue. If this type of issue occurs during the application of a large patch, you may want to be able to restart the original patch from the point of failure. MetaLink Note 175485.1 provides details for applying a patch with adpatch already running.
Database Patching
Database patching consists of either upgrades or interim fixes. Database upgrades are typically complex in nature and require installation of new software when upgrading from one point release to another. Obsolete and new initialization parameters must be reviewed when upgrading to a new release of the database.
Database upgrades can be accomplished manually or by using dbmig, the database migration utility. Since the method for upgrading the database is version and platform dependent, the associated readme file for the upgrade must be reviewed, and the steps required to perform the upgrade should be documented.
Interim patch fixes for the database are applied as the owner of the database install with the opatch utility or by running an operating system script. Details on how to apply database patches are outlined in the patch’s readme.
Before upgrading or applying a patch to the database, the oraInst.loc file must point to the correct Oracle inventory location for the database ORACLE_HOME. It is also important to cleanly shut down the database before proceeding, and to perform a cold database backup.
The opatch utility is downloaded from MetaLink as patch number 2617419. The opatch utility requires Perl and JDK to function, and they must be installed and specified in the path and library environment variables. Once the opatch utility has been downloaded and unbundled, the Opatch directory of the opatch unbundled patch should be added to the PATH, as in the following example:
$export PATH=$PATH:/[path_of_2617419]/Opatch
The library path of Perl must also be specified with the following PERL5LIB environment variable, as in the following example:
$export PERL5LIB=[path_of_PERL]/lib
To validate that opatch is functioning properly, execute the following command with the lsinventory option:
$opatch lsinventory
Once opatch has been successfully set up, the database interim patch fix may be applied. To do this, first review the readme file for the patch. Make certain that all prerequisites have been met. Document any post-patching steps that are required. Download the patch and unbundle it. Change to the directory where the patch has been unbundled. Verify that the database has been shut down. Apply the patch by executing opatch as the database owner with the apply parameter, as in the following example:
$opatch apply
To verify that a patch has successfully been applied, the lsinventory option can again be executed. This will display all patches that have been applied to the database.
Note: If the opatch fails, there may be a patch_locked file located under the hidden directory $ORACLE_HOME/.patch_storage. The opatch utility may not be executed until the patch_locked file is removed.
