Friday, January 2, 2009

Cloning of 11i (11.5.8) from Single-Node to Multi-Node

Introduction.

This paper presents a step-by-step approach for completely cloning an Oracle Applications 11i (AutoConfig enabled environment) using Rapid Clone from one to three nodes, it includes Port Selection, Forms Server, Reports Server, Apache Server and Concurrent Processing, also all the scripts or programs which are used to startup up and shutdown all the services. The cloning word means to do a functional copy of an existing environment.

Some of the reasons to do a cloning process are:

• To create a test environment from an existing production environment to test some patches or to reproduce any production issues.
• To keep a test environment with the most current infomation of a production environment.
• To move any existing environment to other servers.

Just copying the application directories doesn’t mean that our new environment will work properly, we need to do some additional steps or tasks to have a functional environment.

Rapid Install 11.5.8 version creates an Oracle Applications environment which is compatible with “Rapid Clone”, by the way Oracle always recommends to apply the latest Rapid Clone’s patches, for example I applied patch 3130740, “Patch 11i.ADX.D”.

Oracle also recommends to use the latest tool to do a cloning process, so we will use Rapid Clone, by the way to do a clone from multinode to singlenode installation is still not supported by Oracle, but I have heard some partners have done that but using the manual method (No Rapid Clone), the most important thing here and I have done is to keep a single node installation with same patch level than a multinode installation to be able to refresh my production environment (multinode installation) to my test environment (single node installation).

The standard method to do a Rapid Clone cloning procedure follows the next steps:

• To prepare the source environment to be able to use Rapid Clone method (some patches must be needed).
• Cloning the Oracle Applications environment.
• Copying the source environment.
• Configuring the target environment.
• To make the post-cloning tasks.
Source System Configuration.

In our source system, we made a single node installation, in this server we installed Oracle RDBMS software 8.1.7.4 version, Forms, Reports and Apache Servers, also the Concurrent Processing tier.
Table 1. Source System Configuration

System Configuration Single-node, one server
Database, Administration, Concurrent Processing Server Forms, Reports and Apache Servers. Sun Solaris 5.8 64 bit
Platform Sun Solaris 5.8 64-bit
Database Version 8.1.7.4
Application Version 11.5.8
Forms Client Connectivity Jinitiator (1.1.8.16)
User Interface Forms, HTML/Javascript (Self-service Web)
Processors 4 at 750 Mhz.
RAM Memory 8 Gbytes

Table 2. Target System Configuration

System Configuration Multi-node, three servers
Database Server and Administration Server. Sun Solaris 5.8 64 bit
Processors 10 at 900 Mhz.
RAM Memory 20 Gbytes
Forms and Apache Servers. Sun Solaris 5.8 64 bit
Processors 8 at 900 Mhz.
RAM Memory 18 Gbytes
Reports and Concurrent Processing Servers. Sun Solaris 5.8 64 bit
Processors 4 at 900 Mhz.
RAM Memory 8 Gbytes
Database Version 8.1.7.4
Application Version 11.5.8.0.0
Forms Client Connectivity Jinitiator (1.1.8.16)
User Interface Forms, HTML/Javascript (Self-service Web)

Components of 11i architecture.

The components of 11i architecture have changed considerably. The stack includes two distinct interfaces - the “traditional” Forms (ERP applications) interface and Web Interface (Self Service, CRM and Workflow).

The next components of the technology stack are installed:

1. Database Server: This is where the database is hosted, also where Oracle RDBMS is installed. The 8.1.7.4 version is installed by the default for a 11.5.8 version installation. The Ebusiness software will install two ORACLE_HOMES, one for the database and the another one for the Forms, Apache and Report Servers.

2. Forms Server: 11.5.8 Ebusiness software will install Developer 6i version (patchset 9), which includes 6.0.8.18 Forms Server version.

3. Reports Server: 11.5.8 Ebusiness software will install 6.0.8.18 version for Reports and Graphics.

4. Apache Server: The 1.3.19 version of Apache Server is shipped with Ebusiness software 11.5.8, which also install the 1.0.2.2.2 version of Internet Application Server.

5. Concurrent Processing Server: These processes have to execute all the jobs or requests submitted by the Oracle Applications software.

6. Administration Server: Each of the other tech stacks needs an administration component in order to perform tasks such as patching the stack, re-linking executables and re-creating Java files. As a consequence, an Administration component is present on the database server (you have the option to install or not to install this option), Concurrent Manager server, Forms server and Web server. The utilities adpatch and adadmin are part of the Administration tech stack.

7. JInitiator: This Browser Plug in (version 1.1.8.16) is installed on the client, which in most cases is a PC.

Backup Methodology.

Is very important to have a valid backup, it is in case that you need to copy all the application stack from a restore of an application’s backup, also you have the option to do the copy directly from the server where all the technology stack is installed. Database backup could be done online or offline, all depending of the system availability, if you are cloning from an online backup, you will need to recover your database (you will apply the archivelogs until you want, before continuing with the application tier script execution).

S.O. users.

In our initial setup the same user was used to own the database and application stack files. This Unix userid “citcrp3” belongs to the “dba” group. In case you are restoring the files from a tape it is very important to keep the files and privileges like they were taken, so it is recommended to do the restore operation like root user.

Cloning procedure.

1. Preparing the source environment to be able to use Rapid Clone tool.

If you have any prior version to 11.5.7 which is not enabled with AutoConfig tool, you still have the option to use Rapid Clone, by the way you will need first apply all the patches to enable the AutoConfig tool, after that you will need to apply an additional patch to use Rapid Clone method.

In the next table all the prerequisites are listed:

Table 3. Required versions

Software Minimum versión Location Details
Oracle Universal Installer 2.1.0.17 All source system nodes. This version is included in 11.5.7 and above versions, if you have any environment prior to this one, you will need apply patch 2949808.
Perl 5.005 Source and target nodes. Use the Perl shipped with iAS1022 and RDBMS 9i if available or download it from Perl.com. Perl must be in the PATH before cloning. Windows users: do not use Perl from MKSToolKit.
JRE 1.1.8 Source database node. If the RDBMS ORACLE_HOME was not installed using Rapid Install, install JRE 1.1.8 into the /jre/1.1.8 directory.
JRE 1.3.1 Source database node. Windows users only: install JRE 1.3.1 into the /jre/1.3.1 directory.
JDK 1.3.1 Target middle-tier applications nodes. Refer to note 130091.1 of Metalink.
Zip 2.3 All source nodes. Download from InfoZip. Zip must be in your PATH for cloning.

a) For Windows users only: apply patch 2237858 to enable long file names support.

b) Migrate to AutoConfig
If you any prior version to 11.5.7, you must follow all the steps in Note 165195.1 of Metalink to enable the AutoConfig option.

c) Apply additional patches
Verify that 2.3 version of ZIP is included in the $PATH environment variable of the user which will be used to do the cloning prodecure.

Apply patch (2716534) of the Context Editor tool.
Patch 2873456 is a replacement for patch 2716534 until now Feb-19-2004.

Apply patch (2614231), which contains 2.1.0.17.0 version of the OUI.
Installation example:
unzip -d /oui21 p2614231_11i_.zip

Like owner of the database files:
unzip -d /oui21 p2614231_11i_.zip

Note: The above OUI version is installed in 11.5.7 and above versions.

If your enviroment is AutoConfig enabled, you will need to apply the last patch for Rapid Install 3130740, “Patch 11i.ADX.D”, this patch includes Rapid Clone patch, it is to be able to clone Oracle Applications with 8i, 9.0.1 and 9.2.0 versions.

d) Update to JDK 1.3.1
If Oracle Applications uses a prior version to 1.3.1, you will need to update to 1.3.1, you can follow the steps in Note: 212005.1

e) Generate the Applications Context files in the database tier.
You must generate the “xml” file in the directory of the database (RDBMS ORACLE_HOME) /appsutil/.xml and other one for the APPL_TOP directory, if any file doesn’t exist, you can create it with the next command:

a) You must be connected with the owner of the file system of the application if it is a single user installation, or if it is multiuser installation, you must be connected like root user.
b) Start the Context Creation tool in the database tier node (Database Server):

UNIX:
$/appsutil/editcontext/createcontext

f) Update the Applications Context File

UNIX:
$/bin/adconfig.sh contextfile= appspass=
where = $APPL_TOP/admin/SID.xml SID=PROD3 (in our environment).

g) Install the Rapid Clone files in the database server node

Follow the specified procedure in 4 section, step 1 of note 165195.1of Metalink.
Copying all the files of AutoConfig to the RDBMS ORACLE_HOME
• In the Application Tier server, go to the APPL_TOP directory and run the file $APPL_TOP/APPSORA.env to define all the Application’s environment variables.
• Creating the appsutil.zip file.
UNIX:
$perl <$AD_TOP>/bin/admkappsutil.pl, this command will create the appsutil.zip file in the $APPL_TOP/admin/out path.
• In the database server (Database Tier), you will need to copy the appsutil.zip file generated in the above step to the directory.
UNIX:
$cp –p $APPL_TOP/admin/SID/out/appsutil.zip $/
$cd $
$unzip –o appsutil.zip

The –o flag means that all the existing files will be overwritten, if this option isn’t specified, answer with [A]ll.

The last step updates the Rapid Clone files in the directory, including all the generated files after the Rapid Install and AutoConfig patches installation.

Due to adpreclone.pl file isn’t created with just following the above steps, it is necessary to execute all the specified steps in section 4 of Metalink’s document 165195.1, which are detailed in the next steps:

• Generate the Context File in the Database node, executing the next command:

UNIX:
$cd $
$. .env Note: You mustn’t have the Apps environment variables defined.
$cd /appsutil/bin
$adbldxml.sh tier=db appsuser= appspasswd=

Complete the following AutoConfig steps.
Review Prior Manual Configuration Changes.
The Database Context File may not include manual post-install configuration changes made after the Rapid Install completed. Before running the AutoConfig portion of this patch, review any modifications to specific configuration files and reconcile them with the Database Context file.
Note: Prior modifications include any changes made to configuration files as instructed in patch READMEs or other accompanying documents.

Appendix B of Metalink note 165195.1 describes how to review and resolve manual configuration changes.
Generate and Apply AutoConfig Configuration Files.
Attention: This step performs the conversion to AutoConfig. Once completed, the previous configuration will not be available.

Attention: The database server must remain available during the AutoConfig run. All the other database tier services should be shut down.
1. Execute the following commands:
o On UNIX
cd /appsutil/bin
adconfig.sh contextfile= appspass=
o On Windows
cd /d \appsutil\bin
adconfig.cmd contextfile= appspass=
Warning: Running AutoConfig on the database node will update the RDBMS init.ora and network listener files. Be sure to review the configuration changes from step 3. The new AutoConfig files support the use of IFILEs to allow for values to be customized or added as needed.


2. Cloning the Oracle Applications environment.

Use Rapid Clone to create template files for cloning on the source system. After the source system is copied to the target, Rapid Clone updates these templates to contain the new target system configuration settings. Rapid Clone will not change the source system configuration.
There are three phases to the cloning process:

1. Prepare the Source System.
Execute the following commands to prepare the source system for cloning.

a) Prepare the source system database for cloning.

Log on to the source system as the owner of the RDBMS software and run the following commands:

$cd /appsutil/scripts/
where is the SID of the source environment.

$perl adpreclone.pl dbTier

After executing the above command, the $/appsutil/clone directory will be created.

b) Prepare the source system application tier for cloning.

Log on the server as the owner of the applications file (in our environment we used the same user) and execute the next commands in every server where an APPL_TOP exists:

$cd /admin/scripts/
where is the SID of the source environment.

$perl adpreclone.pl appsTier

After executing the above command, the $/clone directory will be created.


2. Copy the Source System to the Target System.

The next table shows all the directories which were copied to target server:


Table 4. Source System directories

Directories: Files description: Used space:
/prod02/prodcrp3/prod3appl APPL_TOP 18.82 Gbytes.
/prod05/prodcrp3/prod3comn COMMON_TOP 1.51 Gbytes.
/prod05/prodcrp3/prod3db ORACLE_HOME for 8.1.7.4 database. 1.99 Gbytes.
/prod08/prodcrp3/prod3ora/8.0.6 ORACLE_HOME for apps tech stack 8.0.6 version 2.93 Gbytes.
/prod08/prodcrp3/prod3ora/iAS Directory for iAS product. 2.80 Gbytes.
/prod01/prodcrp3/prod3data/ Directory where SYSTEM and TEMPORAL datafiles, redologs and controlfiles are stored. 10.04 Gbytes.
/prod03/prodcrp3/prod3data/ Directory for table tablespaces, redologs (group 3 and 4). 10.35 Gbytes.
/prod06/prodcrp3/prod3data/ Directory for index tablespaces, redologs (group 3 y 4). 12.07 Gbytes.
/prod08/prodcrp3/prod3data/ Directory for table tablespaces. 2.38 Gbytes.
Total Used Space: 62.89 Gbytes.

We copied all the files in the database target server node, it is because we had enough space and also for future cloning requirements. Our source system is proesa1880 server and our database production target node name is gpdb. Like example, the next commands were executed in gpdb server to copy the required files:

rcp -pr proesa1880:/prod01/prodcrp3/prod3data /prod13/archives/PROD3-BACKUP/prod01/prodcrp3/
rcp -pr proesa1880:/prod02/prodcrp3/prod3appl /prod13/archives/PROD3-BACKUP/prod02/prodcrp3/
rcp -pr proesa1880:/prod03/prodcrp3/prod3data /prod13/archives/PROD3-BACKUP/prod03/prodcrp3/
rcp -pr proesa1880:/prod05/prodcrp3/prod3comn /prod12/PROD3-BACKUP/prod05/prodcrp3/
rcp -pr proesa1880:/prod05/prodcrp3/prod3db /prod12/PROD3-BACKUP/prod05/prodcrp3/
rcp -pr proesa1880:/prod06/prodcrp3/prod3data /prod13/archives/PROD3-BACKUP/prod06/prodcrp3/
rcp -pr proesa1880:/prod08/prodcrp3/prod3data /prod12/PROD3-BACKUP/prod08/prodcrp3/
rcp -pr proesa1880:/prod08/prodcrp3/prod3ora /prod12/PROD3-BACKUP/prod08/prodcrp3/
rcp –p proesa1880:/prod01/prodcrp3/prod3data/temp01.dbf /prod12/PROD13-BACKUP/prod01/prodcrp3/prod3data/


a) Copy the application tier file system.

Log on to the source system application tier nodes as the owner of the apps files and do the next tasks:
-Shut down the application tier server processes.
-Copy the following application tier directories from source node to the target application node:





<806 ORACLE_HOME>


The next table shows the directories which were copied from source server (singlenode) to target servers (multinode).

Table 5. Source System directories

Application Tiers
Directory name: Source Directory in proesa1880 server: Target directory in gpapps server: Target directory in gprep server:
/prod02/prodcrp3/prod3appl /prod01/oraprod/prodappl /prod01/oraprod/prodappl
/prod05/prodcrp3/prod3comn/html /prod02/oraprod/prodcomn/html /prod02/oraprod/prodcomn/html
/prod05/prodcrp3/prod3comn/java /prod02/oraprod/prodcomn/java /prod02/oraprod/prodcomn/java
/util /prod05/prodcrp3/prod3comn/util /prod02/oraprod/prodcomn/util /prod01/oraprod/prodcomn/util
/clone /prod05/prodcrp3/prod3comn/clone /prod01/oraprod/prodcomn/clone /prod01/oraprod/prodcomn/clone
<806 ORACLE_HOME> /prod08/prodcrp3/prod3ora/8.0.6 /prod03/oraprod/prodora/8.0.6 /prod03/oraprod/prodora/8.0.6
/prod08/prodcrp3/prod3ora/iAS /prod03/oraprod/prodora/iAS /prod03/oraprod/prodora/iAS


Like I said above in this document, all the directories were copied to database server node (gpdb), after that the application directories were copied to two nodes (gpapps and gprep servers). The next commands were used to copy the application directories, they were executed from gpapps and gprep servers (application tier nodes).

rcp -pr gpdb:/prod13/archives/PROD3-BACKUP/prod02/prodcrp3/ /prod01/oraprod/
rcp -pr gpdb:/prod12/PROD3-BACKUP/prod05/prodcrp3/prod3comn/html /prod02/oraprod/prodcomn/
rcp -pr gpdb:/prod12/PROD3-BACKUP/prod05/prodcrp3/prod3comn/java /prod02/oraprod/prodcomn/
rcp -pr gpdb:/prod12/PROD3-BACKUP/prod05/prodcrp3/prod3comn/util /prod02/oraprod/prodcomn/
rcp -pr gpdb:/prod12/PROD3-BACKUP/prod05/prodcrp3/prod3comn/clone /prod02/oraprod/prodcomn/
rcp -pr gpdb:/prod12/PROD3-BACKUP/prod05/prodcrp3/prod3ora/8.0.6 /prod03/oraprod/prodora/
rcp -pr gpdb:/prod12/PROD3-BACKUP/prod05/prodcrp3/prod3ora/iAS /prod03/oraprod/prodora/


b) Copy the database tier file system.

Log on to the source system database node as the ORACLE user.
-Perform a normal shutdown of the source system database.
-Copy the source system database files to the target system.

The next table shows the directories which were copied from source server (singlenode) to target servers (multinode).


Table 6. Source System database directories

Description: Source files and directories: Target files and directories:
Temporary datafiles /prod01/prodcrp3/prod3data/temp01.dbf
/prod06/prodcrp3/prod3data/temp02.dbf /prod02/oraprod/proddata/temp01.dbf
/prod02/oraprod/proddata/temp02.dbf
System datafiles /prod01/prodcrp3/prod3data/system09.dbf
/prod01/prodcrp3/prod3data/system08.dbf
/prod01/prodcrp3/prod3data/system07.dbf
/prod01/prodcrp3/prod3data/system06.dbf
/prod01/prodcrp3/prod3data/system05.dbf
/prod01/prodcrp3/prod3data/system04.dbf
/prod01/prodcrp3/prod3data/system03.dbf
/prod01/prodcrp3/prod3data/system02.dbf
/prod01/prodcrp3/prod3data/system01.dbf /prod03/oraprod/proddata/ system09.dbf
/prod03/oraprod/proddata/ system08.dbf
/prod03/oraprod/proddata/ system07.dbf
/prod03/oraprod/proddata/ system06.dbf
/prod03/oraprod/proddata/ system05.dbf
/prod03/oraprod/proddata/ system04.dbf
/prod03/oraprod/proddata/ system03.dbf
/prod03/oraprod/proddata/ system02.dbf
/prod03/oraprod/proddata/ system01.dbf
Rollback datafiles /prod03/prodcrp3/prod3data/rbs01.dbf
/prod03/prodcrp3/prod3data/rbs02.dbf /prod02/oraprod/proddata/ rbs01.dbf
/prod02/oraprod/proddata/ rbs02.dbf
Table Tablespaces /prod03/prodcrp3/prod3data/
/prod08/prodcrp3/prod3data/ /prod01/oraprod/proddata/*
/prod01/oraprod/proddata/*
Index tablespaces /prod06/prodcrp3/prod3data/ /prod02/oraprod/proddata/*


After copying the source database files to the target database node, you can startup all the source services including the application services.


3. Configure the Target System.

In this section, I show all the output of the executed commands. This section is divided in two parts, one for the Database Tier and the other one for the Application Tiers.

a) Configuring the target system database server.

Execute the next commands like the owner of the target database server files, in our environment it was called “oraprod”. Also don’t forget to add the /usr/ccs/bin value to the PATH environment variable, it is needed to execute the perl script in this node.

$cd /appsutil/clone/bin
$perl adcfgclone.pl dbTier

After executing the above command, the next options are showed on the screen:


Please enter the APPS password:
apps

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:

Provide the values required for creation of the new Database Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:

Target system database SID [PROD3]:PROD

Target system domain name []: ommited by security reasons

Target system RDBMS ORACLE_HOME directory [/prod05/prodcrp3/prod3db/8.1.7]:/prod11/oraprod/proddb/8.1.7

Target system utl_file accessible directories list [/usr/prod3]:/usr/prod

Number of DATA_TOP's on the target system [4]:

Target system DATA_TOP 1 [/prod03/prodcrp3/prod3data]:/prod03/oraprod/proddata

Note: This is the used path for “SYSTEM” datafiles.

Target system DATA_TOP 2 [/prod01/prodcrp3/prod3data]:/prod01/oraprod/proddata

Note: This is the used path for “DATA” datafiles.

Target system DATA_TOP 3 [/prod08/prodcrp3/prod3data]:/prod02/oraprod/proddata

Note: This is the used path for “INDEX”, “TEMPORARY” and “ROLLBACK” datafiles.

Target system DATA_TOP 4 [/prod06/prodcrp3/prod3data]:/prod06/oraprod/proddata

The new context file has to contain port numbers related to target system.

Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:

Database port is 1525

Backing up /prod11/oraprod/proddb/8.1.7/appsutil/PROD_gpdb.xml to /prod11/oraprod/proddb/8.1.7/appsutil/PROD_gpdb.xml0.bak

Creating the new Database Context file from :
/prod11/oraprod/proddb/8.1.7/appsutil/template/adxdbctx.tmp

The new database context file has been created :
/prod11/oraprod/proddb/8.1.7/appsutil/PROD_gpdb.xml

Log file located at /tmp/CloneContext_11191007.log

Running Rapid Clone with command:
perl ./adclone.pl java=./../jre/1.3.1 mode=apply stage=/prod11/oraprod/proddb/8.1.7/appsutil/clone component=dbTier method=CUSTOMdbctxtg=/prod11/oraprod/proddb/8.1.7/appsutil/PROD_gpdb.xml showProgress

Using adconfig.zip : 115.340

Beginning database tier Apply - Wed Nov 19 22:09:37 2003
Log file located at /prod11/oraprod/proddb/8.1.7/appsutil/log/PROD_gpdb/ApplyDBTier_111
91009.log
/ 4% completed
\
| 13% completed
- 19% completed
|
/ 29% completed
\ 29% completed
/
- 29% completed
| 29% completed
-
\ 29% completed
/ 29% completed
\
| 29% completed
- 29%

/ 29% completed
\ 29%

- 29% completed
| 29

\ 29% completed

Completed Apply...
Wed Nov 19 22:13:08 2003

Beginning APPSDB_PROD registration to central inventory...

ORACLE_HOME NAME : APPSDB_PROD
ORACLE_HOME PATH : /prod11/oraprod/proddb/8.1.7
INVENTORY LOCATION : /var/opt/oracle/oraInst.loc

Log file located at /u01/oraprod/oraInventory/logs/OracleHomeCloner_11191013.log
RC-00123: ORACLE_HOME APPSDB_PROD already exists inside the inventory

Starting database listener for PROD:
Running:
/prod11/oraprod/proddb/8.1.7/appsutil/scripts/PROD_gpdb/addlnctl.sh start PROD

You are running addlnctl.sh version 115.3


Starting listener process PROD ...


Listener PROD has already been started.

addlnctl.sh: exiting with status 0


b) Configuring the target system application tier server nodes.

Execute the next commands like the owner of the target application server files, in our environment we used the same user than the database server and it was called “oraprod”. Also don’t forget to add the /usr/ccs/bin value to the PATH environment variable, it is needed to execute the perl script in this node.

$cd /clone/bin
$perl adcfgclone.pl appsTier

After executing the above command, the next options are showed on the screen:

Please enter the APPS password:
prodfrut

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:


Provide the values required for creation of the new APPL_TOP Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:

Target system database SID [PROD3]:PROD

Target system domain name []: ommited by security reasons

Target system database server node [proesa1880]:gpdb

Does the target system have more than one application tier server node (y/n) [n] ?:y

Target system concurrent processing node [proesa1880]:gprep

Target system administration node [proesa1880]:gpapps

Target system forms server node [proesa1880]:gpapps

Target system web server node [proesa1880]:gpapps

Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:

Target system APPL_TOP mount point[/prod02/prodcrp3/prod3appl]:
/prod01/oraprod/prodappl

Target system COMMON_TOP directory [/prod05/prodcrp3/prod3comn]:
/prod02/oraprod/prodcomn

Target system 8.0.6 ORACLE_HOME directory [/prod08/prodcrp3/prod3ora/8.0.6]:
/prod03/oraprod/prodora/8.0.6

Target system iAS ORACLE_HOME directory [/prod08/prodcrp3/prod3ora/iAS]:
/prod03/oraprod/prodora/iAS

Location of JDK 1.3.1 on the target system [/usr/j2se]:

The new context file has to contain port numbers related to target system.

Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:
Web Listener port is 8004
Complete port information available at /prod01/oraprod/prodappl/admin/out/PROD_gpapps/portpool.lst

UTL_FILE_DIR on database tier consists of the following directories.

1. /usr/prod -- This directory must exist also the user must have the needed privileges.
2. /prod11/oraprod/proddb/8.1.7/appsutil/outbound/PROD_gpdb
Choose a value which will be set as APPLPTMP value on the target node [1]:

Creating the new APPL_TOP Context file from :
/prod01/oraprod/prodappl/ad/11.5.0/admin/template/adxmlctx.tmp

The new APPL_TOP context file has been created :
/prod01/oraprod/prodappl/admin/PROD_gpapps.xml

Log file located at /tmp/CloneContext_11191134.log

Running Rapid Clone with command:
perl ./adclone.pl java=./../jre/1.3.1 mode=apply stage=/prod02/oraprod/prodcomn/
clone component=appsTier method=CUSTOM appctxtg=/prod01/oraprod/prodappl/admin/PROD_gpapps.xml showProgress

Using adconfig.zip : 115.340

Beginning application tier Apply - Wed Nov 19 23:36:51 2003
Log file located at /prod01/oraprod/prodappl/admin/PROD_gpapps/log/ApplyAppsTier_11191136.log

Completed Apply...
Wed Nov 19 23:48:36 2003

Beginning APPSIAS_PROD registration to central inventory...

ORACLE_HOME NAME : APPSIAS_PROD
ORACLE_HOME PATH : /prod03/oraprod/prodora/iAS
INVENTORY LOCATION : /var/opt/oracle/oraInst.loc

Log file located at /u02/oratest/testora/oraInventory/logs/OracleHomeCloner_11191148.log
RC-00123: ORACLE_HOME APPSIAS_PROD already exists inside the inventory

Starting application Services for PROD:
Running:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gpapps/adstrtal.sh APPS/

You are running adstrtal.sh version 115.11

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gpapps/adapcctl.sh start
script returned:
****************************************************

adapcctl.sh version 115.34

Starting Apache Web Server Listener (dedicated HTTP) ...
Starting Apache Web Server Listener (dedicated PLSQL) ...
/prod03/oraprod/prodora/iAS/Apache/Apache/bin/apachectl start: httpd started

adapcctl.sh: exiting with status 0

.end std out.

****************************************************

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gpapps/adalnctl.sh start
script returned:
****************************************************

adalnctl.sh version

Checking for FNDFS executable.
Starting listener process APPS_PROD.

adalnctl.sh: exiting with status 0

.end std out.

****************************************************

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gpapps/adfrmctl.sh start
script returned:
****************************************************

You are running adfrmctl.sh version 115.29

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Starting forms server for PROD on port 9004.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

adfrmctl.sh: exiting with status 0

.end std out.

****************************************************

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gpapps/adfmcctl.sh start
script returned:
****************************************************

You are running adfmcctl.sh version 115.16

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Starting forms load balancing client for PROD.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

adfmcctl.sh: exiting with status 0

.end std out.
****************************************************

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gpapps/adfmsctl.sh start
script returned:
****************************************************

You are running adfmsctl.sh version 115.12

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
starting forms metrics server for PROD.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

adfmsctl.sh: exiting with status 0

.end std out.
****************************************************

Executing service control script:

/prod02/oraprod/prodcomn/admin/scripts/PROD_gpapps/jtffmctl.sh start
script returned:
****************************************************

You are running jtffmctl.sh version 115.5

Starting Fulfillment Server for PROD on port 9304 ...

jtffmctl.sh: exiting with status 0

.end std out.
****************************************************

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gpapps/addisctl.sh start
script returned:
****************************************************

addisctl.sh version 115.8

/prod03/oraprod/prodora/8.0.6/vbroker/bin/osagent
Started osagent.
Osagent logs messages to the file /prod03/oraprod/prodora/8.0.6/discwb4/util/osagent.log
.
Waiting for OAD to start...
Started OAD.
OAD logs messages to the file /prod03/oraprod/prodora/8.0.6/discwb4/util/oad.log.
Discoverer Locator Started.
Locator logs messages to the file /prod03/oraprod/prodora/8.0.6/discwb4/util/locator.log
.
Registering Discoverer Session
Completed registration of repository_id = IDL:DCISessionManager:1.0
object_name = _8004OracleDiscovererSession4
reference data =
path_name = /prod03/oraprod/prodora/8.0.6/discwb4/util/runses.sh
activation_policy = UNSHARED_SERVER
args = (length=4)[-session; _8004OracleDiscovererSe
ssion4; -preference; _8004OracleDiscovererPreferences4; ]
env = NONE
for OAD on host
Registering the Collector
Completed registration of repository_id = IDL:DCICollector:1.0
object_name = _8004OracleDiscovererCollector4
reference data =
path_name = /prod03/oraprod/prodora/8.0.6/discwb4/util/runcol.sh
activation_policy = SHARED_SERVER
args = (length=2)[-collector; _8004OracleDiscoverer
Collector4; ]
env = NONE
for OAD on host
Applying preferences from file : /prod03/oraprod/prodora/8.0.6/discwb4/util/pref.txt
Finished applying preferences

Closing down registry..
Registry File sync...
Registering Discoverer Preference Repository
Completed registration of repository_id = IDL:DCICORBAInterface:1.0
object_name = _8004OracleDiscovererPreferences4
reference data =
path_name = /prod03/oraprod/prodora/8.0.6/discwb4/util/runpref.sh
activation_policy = SHARED_SERVER
args = (length=2)[-preference; _8004OracleDiscovere
rPreferences4; ]
env = NONE
for OAD on host
addisctl.sh: exiting with status 0

.end std out.
****************************************************

All enabled services for this node are started.
Check logfile /prod02/oraprod/prodcomn/admin/log/PROD_gpapps/11192348.log for details

Exiting with status 0

If you look there are some warning messages, it is because my application tier password didn’t match with my database server execution script for the APPS user password.

The next output is for my Concurrent Processing server (in our environment we are using two nodes for the application tier).

Execute the next commands like the owner of the target application server files, in our environment we used the same user than the database server and it was called “oraprod”. Also don’t forget to add the /usr/ccs/bin value to the PATH environment variable, it is needed to execute the perl script in this node.

$cd /clone/bin
$perl adcfgclone.pl appsTier

After executing the above command, the next options are showed on the screen:

Please enter the APPS password:
prodfrut

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:

Provide the values required for creation of the new APPL_TOP Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:

Target system database SID [PROD3]:PROD

Target system domain name []: ommited by security reasons

Target system database server node [proesa1880]:gpdb

Does the target system have more than one application tier server node (y/n) [n] ?:y

Target system concurrent processing node [proesa1880]:gprep

Target system administration node [proesa1880]:gpapps

Target system forms server node [proesa1880]:gpapps

Target system web server node [proesa1880]:gpapps

Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:

Target system APPL_TOP mount point [/prod02/prodcrp3/prod3appl]:
/prod01/oraprod/prodappl

Target system COMMON_TOP directory [/prod05/prodcrp3/prod3comn]:
/prod02/oraprod/prodcomn

Target system 8.0.6 ORACLE_HOME directory [/prod08/prodcrp3/prod3ora/8.0.6]:
/prod03/oraprod/prodora/8.0.6

Target system iAS ORACLE_HOME directory [/prod08/prodcrp3/prod3ora/iAS]:
/prod03/oraprod/prodora/iAS

Location of JDK 1.3.1 on the target system [/usr/j2se]:

The new context file has to contain port numbers related to target system.

Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:

Web Listener port is 8004
Complete port information available at /prod01/oraprod/prodappl/admin/out/PROD_gprep/portpool.lst

UTL_FILE_DIR on database tier consists of the following directories.

1. /usr/prod -- This directory must exist also the user must have the needed privileges.

2. /prod11/oraprod/proddb/8.1.7/appsutil/outbound/PROD_gpdb
Choose a value which will be set as APPLPTMP value on the target node [1]:

Creating the new APPL_TOP Context file from :
/prod01/oraprod/prodappl/ad/11.5.0/admin/template/adxmlctx.tmp

The new APPL_TOP context file has been created :
/prod01/oraprod/prodappl/admin/PROD_gprep.xml

Log file located at /tmp/CloneContext_11191159.log

Running Rapid Clone with command:
perl ./adclone.pl java=./../jre/1.3.1 mode=apply stage=/prod02/oraprod/prodcomn/
clone component=appsTier method=CUSTOM appctxtg=/prod01/oraprod/prodappl/admin/PROD_gpr
ep.xml showProgress

Using adconfig.zip : 115.340

Beginning application tier Apply - Thu Nov 20 00:01:51 2003
Log file located at /prod01/oraprod/prodappl/admin/PROD_gprep/log/ApplyAppsTier_1120120
1.log

Thu Nov 20 00:13:04 2003

Beginning APPSIAS_PROD registration to central inventory...

ORACLE_HOME NAME : APPSIAS_PROD
ORACLE_HOME PATH : /prod03/oraprod/prodora/iAS
INVENTORY LOCATION : /var/opt/oracle/oraInst.loc

Log file located at /u02/oratest/testora/oraInventory/logs/OracleHomeCloner_11201213.log
RC-00123: ORACLE_HOME APPSIAS_PROD already exists inside the inventory

Starting application Services for PROD:
Running:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gprep/adstrtal.sh APPS/

You are running adstrtal.sh version 115.11

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gprep/adalnctl.sh start
script returned:
****************************************************

adalnctl.sh version

Checking for FNDFS executable.
Starting listener process APPS_PROD.

adalnctl.sh: exiting with status 0

.end std out.
****************************************************

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gprep/adcmctl.sh start
script returned:
****************************************************

You are running adcmctl.sh version 115.17

Starting concurrent manager for PROD ...
Starting PROD_1120@PROD Internal Concurrent Manager
Default printer is noprint

adcmctl.sh: exiting with status 0

.end std out.
****************************************************

Executing service control script:
/prod02/oraprod/prodcomn/admin/scripts/PROD_gprep/adrepctl.sh start
script returned:
****************************************************

You are running adrepctl.sh version 115.16

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
starting Reports Server for PROD on port 7004.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

adrepctl.sh: exiting with status 0

.end std out.
****************************************************

All enabled services for this node are started.
Check logfile /prod02/oraprod/prodcomn/admin/log/PROD_gprep/11200013.log for details

Exiting with status 0

If you look there are some warning messages, it is because my application tier password didn’t match with my database server execution script for the APPS user password.

At this moment all the services must be running properly, by the way it is very important to perform all the finishing tasks which are specified in the next section.

3. Finishing tasks.

This section lists tasks that may be necessary depending on your implementation and the intended use of the cloned system.

a) Update profile options: Rapid Clone updates only site level profile options. If any other profile options are set to instance specific values, you must update them manually.

b) Update printer settings: If the new cloned system needs to utilize different printers, update the target system with the new printer settings now.

c) Update workflow configuration settings: Cloning an Oracle Applications instance will not update the host and instance specific information used by Oracle Workflow. Review the following tables and columns to verify there is no instance specific data in the Workflow configuration on the target system:

Tabla 7. Target System tables with values to be modified.

Nombre de la tabla: Nombre de la columna: Valores detallados de la columna:
WF_NOTICATION_ATTRIBUTES TEXT_VALUE Value starts with http:// : Update to new web host.
WF_ITEM_ATTRIBUTE_VALUES TEXT_VALUE Value starts with "http:// : Update to new web host.
WF_SYSTEMS GUID Create a new system defined as the new global database name using the Workflow Administrator Web Applications responsibility.
WF_SYSTEMS NAME Value needs to be replaced with the database global name
WF_AGENTS ADDRESS Update database link with the new database global name.
FND_FORM_FUNCTIONS WEB_HOST_NAME Update with the new web host name.
FND_FORM_FUNCTIONS WEB_AGENT_NAME Update to point at the new PLSQL listener name.
FND_CONCURRENT_REQUESTS LOGFILE_NODE_NAME *Update with the new value of the Concurrent Processing server name.
FND_CONCURRENT_REQUESTS OUTFILE_NODE_NAME *Update with the new value of the Concurrent Processing server name.
FND_CONCURRENT_REQUESTS LOGFILE_NAME *Update to the new path of the concurrent processing server name.
FND_CONCURRENT_REQUESTS OUTFILE_NAME *Update to the new path of the concurrent processing server name.
* See the below steps.

To update the value of the new Concurrent Processing server, I used the next sentences:

UPDATE FND_CONCURRENT_REQUESTS
SET LOGFILE_NODE_NAME = 'GPREP',
OUTFILE_NODE_NAME = 'GPREP';

COMMIT;

To update the value of the new path for the requests in the FND_CONCURRENT_REQUESTS table, I used the next sentences:

UPDATE FND_CONCURRENT_REQUESTS
SET LOGFILE_NAME = REPLACE(LOGFILE_NAME, '/prod05/prodcrp3/prod3comn/admin/log/PROD3/', '/prod02/oraprod/prodcomn/admin/log/PROD_gprep/'),
OUTFILE_NAME = REPLACE(OUTFILE_NAME, '/prod05/prodcrp3/prod3comn/admin/out/PROD3/', '/prod02/oraprod/prodcomn/admin/out/PROD_gprep/');

COMMIT;
a) After the above changes, it is suggested to complete the next steps to be able to execute the AD utilities (adadmin, adpatch) properly.

It is suggested to rename the topfile.txt which is placed in $APPL_TOP/admin/topfile.txt path, this file will be created after executing any AD utility.

b) Maybe could be necessary to change or just verify the APPS’s password in the $IAS_TOP/Apache/modplsql/wdbsvr.app file (just like me), it is needed just to avoid any privilege problem when accessing the application after the cloning procedure.

c) To be able to execute the “JAR’s” regeneration files option from adadmin utility, it will be necessary to execute the next command to regenerate the new certification key for this new cloned instance:

$adjkey –initialize

d) If you want to change the default colorScheme inside of the application accessed by http://hostname.domain:port/dev60cgi/f60cgi URL, it will be necessary to change the colorScheme parameter in $OA_HTML/bin/appsweb_SID.cfg file, where SID=PROD (for our environment).

e) Avoiding Yellow Bar Warning. We will need to follow the next additional steps to solve the Yellow Warning Bar, because we have just generated a new certification file, we will need to copy the next source files to the target Forms Server node:

$APPL_TOP/admin/appltop.cer this file will be copied to $APPL_TOP/admin/appltop.cer path of the Forms Server target node.
$APPL_TOP/admin/adsign.txt this file will be copied to $APPL_TOP/admin/adsign.txt path of the Forms Server target node.
$HOME/identitydb.obj this file will be copied to $HOME/identitydb.obj path of the owner for the apps files.

After copying these files, it will be necessary to execute the regeneration of JAR’s files twice, choosing the regeneration of files for all the products.

f) It is also recommended to compile all the APPS’s objects, this is done executing the adadmin utility.

Target System Configuration.

The next table shows the final configuration for target system, the directories which were used for all the three nodes: gpdb, gpapps y gprep.

Table 8. Target System Configuration.

Server: Files: Directory: Gbytes size:
gpdb Tables Datafiles /prod01/oraprod/proddata/ 10.58
Index Datafiles /prod02/oraprod/proddata/ 9.93
SYSTEM tablespace’s Datafiles /prod03/oraprod/proddata/ 6.41
Rollback Segments Datafiles /prod08/oraprod/proddata/rbs01.dbf
/prod12/oraprod/proddata/rbs02.dbf 1.00
1.00
Temporary Segments Datafiles /prod10/oraprod/proddata/temp01.dbf
/prod09/oraprod/proddata/temp02.dbf 2.00
2.00
Controlfiles /prod03/oraprod/proddata/cntrl01.dbf
/prod01/oraprod/proddata/cntrl02.dbf
/prod02/oraprod/proddata/cntrl03.dbf 8.4 Mbytes
8.4 Mbytes
8.4 Mbytes
Redolog Files Group # 1:
/prod11/oraprod/proddata/log01a.dbf
/prod05/oraprod/proddata/log01b.dbf
Group # 2:
/prod10/oraprod/proddata/log02a.dbf
/prod04/oraprod/proddata/log02b.dbf
Group # 3:
/prod11/oraprod/proddata/log03a.dbf
/prod05/oraprod/proddata/log03b.dbf
Group # 4:
/prod10/oraprod/proddata/log04a.dbf
/prod04/oraprod/proddata/log04b.dbf 50 Mbytes
50 Mbytes

50 Mbytes
50 Mbytes

50 Mbytes
50 Mbytes

50 Mbytes
50 Mbytes
Archivelogs /prod13/archives/ 50 Gbytes Available
Oracle (RDBMS 8.1.7.4) /prod11/oraprod/proddb/8.1.7 2.03
gpapps APPL_TOP /prod01/oraprod/prodappl 18.41
COMMON_TOP /prod02/oraprod/prodcomn 1.48
ORACLE_HOME /prod03/oraprod/prodora/8.0.6 2.97
iAS_HOME /prod03/oraprod/prodora/iAS 3.58
gprep APPL_TOP /prod01/oraprod/prodappl 18.41
COMMON_TOP /prod02/oraprod/prodcomn 1.48
ORACLE_HOME /prod03/oraprod/prodora/8.0.6 2.97
iAS_HOME /prod03/oraprod/prodora/iAS 3.58

Accessing application.

You can get the logon screen, following any of the next URLs:

1. http://.:8004
2. http://.:8004/dev60cgi/f60cgi

Testing the new functionality.

It is advisable to have the source system, which can be compared against the new target system (cloned system), by the way this document it is not intended to mention every test that you have to follow to confirm that your new cloning procedure has worked as it is intended, it only depends on your own environment, by the way I have just done the next tests:

-Requests execution.
-If you have PDF reports, it is advisable to test the output for this kind of reports, maybe you will need to change the DISPLAY variable incide of the COMMON_TOP’s scripts.
-If you are using Discoverer Viewer and Discoverer Plus try to test their functionality.
-If you are using Oracle Workflow try to test its functionality.
-Oracle Alert can also be tested.

Conclusion.

This document has like main goal to describe the process that was followed to have a successful cloning procedure from 1 to 3 nodes using Rapid Clone method in a 11.5.8 version. I started from point g) Install the Rapid Clone files in the database server node, it is because our environment was AutoConfig enabled. This procedure can be applied in the same way for a 1 to 2 nodes or 1 to 1 node.

Note.

This paper is destined to share my experience and issues with other colleagues. I tried to show all the issues and troubles that I had while making a cloning procedure using Rapid Clone method for first time. This document is not intended to be a must to follow guide, if you have any other comments or issues that you can share they are welcome.

Database general activities and issues II

Continuing the prevous topic with the same heading

Loss of TEMP Datafile
Symptoms: On large sorts (select distinct, order by, group by, union) that can't be done in memory, the sort will fail with "ORA-01157: cannot identify data file 3 - file not found" if the loss happened in the middle of the sort, or "ORA-01116: error in opening database file 3" if the loss happened before the sort started, along with the file name "ORA-01110: data file 3: '/u03/oradata/PROD/temp_PROD_01.dbf'". Nothing is put into the alert.log file, and no trace files will be generated for this.
You may also need to drop and recreate the TEMP tablespace after a database recovery, which may be indicated by an error message like "ORA-25153: temporary tablespace is empty" when you first log into your application (such as SCT Banner), although you probably won't get that message just logging into sqlplus.
Action: All you need to do is to take the datafile offline and drop and recreate the TEMP tablespace, which can be done either while the database is still running (from sqlplus) or beginning at the mount state (when connected as sysdba). Here, we are using a locally managed temporary tablespace.
$ sqlplus "/ as sysdba"
SQL> alter database datafile '/u03/oradata/PROD/temp_PROD_01.dbf' offline;
Then, if from mount state, do an "alter database open;" here.
SQL> select file_name,bytes/1024 kbytes from dba_temp_files;
Shows the size of the TEMP datafile(s) in Kbytes.
SQL> select initial_extent/1024 kbytes from dba_tablespaces
where tablespace_name = 'TEMP';
Shows the uniform size of the extents in Kbytes.
SQL> drop tablespace temp;
SQL> !rm /u03/oradata/PROD/temp_PROD_01.dbf
SQL> create tablespace temp
tempfile '/u03/oradata/PROD/temp_PROD_01.dbf' size 40064K
extent management local uniform size 640K;

TEMP Datafile Offline

Symptoms: Similar to Loss of TEMP Datafile, except for the message "ORA-00376: file 3 cannot be read at this time", along with the file name message.
Action: You can offline the datafile and drop and recreate the tablespace, as above, or you can do a datafile recovery with the database open and bring the datafile online.
SQL> recover automatic datafile '/u03/oradata/PROD/temp_PROD_01.dbf';
SQL> alter database datafile '/u03/oradata/PROD/temp_PROD_01.dbf' online;
Loss of INACTIVE Online Redo Log Group (which had already been archived)
Symptoms: The database crashes when an attempt is made to access that redo log group during a logfile switch. Current users will be kicked out with "ORA-01092: ORACLE instance terminated. Disconnection forced", and users trying to log on will get "ERROR: ORA-03114: not connected to ORACLE" and "ERROR: ORA-00472: PMON process terminated with error". Nothing will be placed in the alert.log file; however, there will be pmon (process monitor), dbwr (database writer), and lgwr (log writer) trace files generated in the background dump dest directory. Check these from the unix command line, since the database is not up to check them from:
$ grep background_dump_dest /u00/oracle/product/v723/dbs/initPROD.ora
Shows the pathname of the bdump directory.
$ cd /u00/oracle/admin/PROD/bdump
$ ls -ltr *.trc
Get the latest trace files in the bdump directory, such as pmon_13612.trc and lgwr_32306.trc (the last ones listed).
$ cat pmon_13612.trc
Shows "ORA-00470: LGWR process terminated with error" (also in dbwr).
$ cat lgwr_32306.trc
Shows "ORA-00313: open failed for members of log group 3 of thread 1" and messages containing the missing log file names like "ORA-00312: online log 3 thread 1: '/u03/oradata/PROD/log_PROD_3B.rdo'".
Action: You will need to startup the database in the mount state, drop the missing logfile group (and remove the group's member files, if they are not already gone), and add the logfile group back, at which time the database can be opened for use.
$ sqlplus "/ as sysdba"
SQL> startup
Shows "ORA-01081: cannot start already-running ORACLE - shut it down first".
SQL> startup force
Or, you could have done a "shutdown abort" followed by a "startup". Shows "Database mounted." and then "ORA-00313: open failed for members of log group 3 of thread 1" with additional messages containing the log file names.
SQL> select bytes/1024 from v$log where group# = 3;
Shows the size in K of the missing group's members.
SQL> select member from v$logfile where group# = 3;
Shows the member (file) names of the missing group.
SQL> alter database drop logfile group 3;
SQL> alter database add logfile group 3
('/u03/oradata/PROD/log_PROD_3A.rdo',
'/u03/oradata/PROD/log_PROD_3B.rdo') size 500K;
Recreates all member files for that group of the given size.
SQL> alter database open;
If just one member of group is lost, the other member(s) will take over and allow Oracle to keep running with no apparent problem. However, alert_PROD.log will show errors such as "ORA-00313: open failed for members of log group 3 of thread 1" and "ORA-00312: online log 3 thread 1: '/u03/oradata/PROD/log_PROD_3A.rdo'" on each log switch for that group. In this case, drop and recreate the member:
SQL> select * from v$log where group# = 3;
If the status is active or current for that group, do an "alter system switch logfile;"
SQL> alter database drop logfile member
'/u03/oradata/PROD/log_PROD_3A.rdo';
SQL> alter database add logfile member
'/u03/oradata/PROD/log_PROD_3A.rdo' to group 3;

Loss of CURRENT Online Redo Log Group (which needed to be archived)

Symptoms: This behaves like the database freezing when the archivelogs disk volume is filled up; however, doing a "df -k" shows that there is plenty of room for more archivelogs. Current users' sessions will freeze, and users trying to log on will get "ERROR: ORA-00257: archiver error. Connect internal only, until freed.", since the archiver process is still waiting to archive that redo log group after Oracle has cycled through the other groups and needs to use that group again. To verify this conclusion:
$ sqlplus "/ as sysdba"
SQL> select value from v$parameter where name = 'background_dump_dest';
Shows the pathname of the bdump directory.
SQL> !tail -200 /u00/oracle/admin/PROD/bdump/alert_PROD.log
Shows "ORA-00286: No members available, or no member contains valid data", indicating that the redo log group is missing or corrupt, as well as messages such as "ORACLE Instance PROD - Can not allocate log, archival required", "Thread 1 cannot allocate new log, sequence 21", and "All online logs needed archiving".
SQL> !ls -ltr /u00/oracle/admin/PROD/bdump/arch*
Get the latest archiver trace file in the bdump directory, such as arch_22882.trc (the last one listed).
SQL> !cat /u00/oracle/admin/PROD/bdump/arch_22882.trc
Also shows "ORA-00286: No members available, or no member contains valid data".
SQL> shutdown abort
SQL> startup
Shows "ORA-00313: open failed for members of log group 2 of thread 1", or whatever group number has the problem.
Then, if you had tried to just drop and recreate the redo log group from this mount state (as in loss of inactive online redo log group), you would have gotten:
SQL> alter database drop logfile group 2;
Fails with "ORA-00350: log 2 of thread 1 needs to be archived".
Action: This requires an incomplete recovery to just before that group was used, since Oracle can't continue without doing a successful archive, and since you can't drop and recreate the redo log group while the archiver has it marked for archiving. Note that all changes in the lost redo log group will be lost (yuck!). An incomplete time-based recovery is shown below:
SQL> connect / as sysdba
SQL> shutdown abort
At this point, restore ALL datafiles from the last backup, any archivelogs that are needed since that time, AND the lost online redo log group's files (this is the only case in which you would restore those redo logs), but, NOT the control files.
SQL> startup mount
SQL> select group#, sequence#, bytes, first_change#, to_char(first_time,'DD-MON-YY HH24:MI:SS'), status from v$log;
Get the first time for changes in that missing group, subtract 1 second from it, and use that value in the recover command below.
SQL> recover automatic database until time '2005-02-14:12:59:59';
SQL> alter database open resetlogs;
Then, immediately shut down the database and make a BACKUP, since the old backup is unuseable after RESETLOGS is used.

Failure During Hot Backup

If you have a failure while you are doing a hot backup on a tablespace, besides doing any recovery that is needed for the particular failure, you will also need to bring those tablespace datafiles back out of hot backup mode. To do this, while the database is in a mount state, do an "end backup" on each of those datafiles before opening the database. This is available in Oracle 7.2 and above - before that, you would have to restore all the datafiles and do an incomplete recovery to the time before the hot backup was started on that tablespace.
$ sqlplus "/ as sysdba"
SQL> startup mount
SQL> select df.name,bk.time from v$datafile df,v$backup bk
where df.file# = bk.file# and bk.status = 'ACTIVE';
Shows the datafiles currently in a hot backup state.
SQL> alter database datafile
'/u03/oradata/PROD/devl_PROD_01.dbf' end backup;
Do an "end backup" on those listed hot backup datafiles.
SQL> alter database open;

Loss of Server Parameter File (spfile)

If your server parameter file (spfile) becomes corrupt, and you haven't been creating a textual init.ora parameter file as a backup, you can pull the parameters from it using the strings command in UNIX to create an init.ora file. You will need to edit the resulting file to get rid of any garbage characters in it (but don't worry about the "*." characters at the beginning of the lines) and make any corrections to it before using it to start your database, but, at least you will have something to go by:
$ cd $ORACLE_HOME/dbs
$ strings spfilePROD.ora >initPROD.ora
If you have been saving off a textual init.ora parameter file as a backup, you can restore that init.ora file to the $ORACLE_HOME/dbs directory in UNIX (or $ORACLE_HOME\database directory in NT). You will need to delete the corrupt spfile before trying to restart your database, since Oracle looks for the spfile first, and the init.ora file last, to use as the parameter file when it starts up the database (or, you could leave the spfile there and use the pfile option in the startup command to point to the init.ora file). Then, once your database is up, you can recreate the spfile using the following (as sysdba):
SQL> create spfile from pfile;