Thursday 16 March 2017

Issues Encountered during Grid Infrastructure and Database Upgrade from 11.2.0.4 to 12.1.0.2

Overview
Upgrading Grid Infrastructure and Database is a very lengthy process and it is obvious that you might encounter issues due to missing prerequisites or BUGS in the upgrade process. If you are lucky or if you meet all the prerequisites before starting the upgrade then it is possible that you might not fall in to issue.

When I upgraded Grid Infrastructure recently from 11.2.0.4 to 12.1.0.2 I encountered an issue which was preventing me from upgrade the Database as a Cluster database. The entire GI upgrade process when smooth, so I can only say that it was a BUG that I encountered.

Error
  • crsctl query crs activeversion show error for node 1.

dm01db01-orcldb1 {/home/oracle}:dcli -g dbs_group -l oracle '/u01/app/12.1.0.2/grid/bin/crsctl query crs activeversion'
dm01db01: kgfnGetFacility: facility=0x22761d8
dm01db01: kgfnInitDiag: diagctx=0x216a100
dm01db01: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db02: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db03: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db04: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db05: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db06: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db07: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db08: Oracle Clusterware active version on the cluster is [12.1.0.2.0]

  • dbua failed with "The database orcldb is a cluster database and cannot be upgrade as the target Oracle Home is for single instance database only". This is due the GI issue above.
  • Review rootupgrade.sh output
I went back and checked my rootupgrade.sh output on node 1. I can see it showed 2 lines which is not expected as part of output.

[root@bs01db01 ~]# /u01/app/12.1.0.2/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.2/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2017/01/10 06:52:50 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2017/01/10 06:52:51 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2017/01/10 06:52:53 CLSRSC-464: Starting retrieval of the cluster configuration data

2017/01/10 06:53:02 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2017/01/10 06:53:02 CLSRSC-363: User ignored prerequisites during installation

2017/01/10 06:53:13 CLSRSC-515: Starting OCR manual backup.

2017/01/10 06:53:15 CLSRSC-516: OCR manual backup successful.

2017/01/10 06:53:19 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2017/01/10 06:53:19 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/11.2.0.4/grid -oldCRSVersion 11.2.0.4.0 -nodeNumber 1 -firstNode true -startRolling true'

ASM configuration upgraded in local node successfully.

2017/01/10 06:53:27 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode

2017/01/10 06:53:27 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2017/01/10 06:53:53 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

OLR initialization - successful
2017/01/10 06:58:48 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
kgfnGetFacility: facility=0x1564768
kgfnInitDiag: diagctx=0x13e4160

CRS-4123: Oracle High Availability Services has been started.
2017/01/10 07:03:18 CLSRSC-472: Attempting to export the OCR

2017/01/10 07:03:18 CLSRSC-482: Running command: 'ocrconfig -upgrade oracle oinstall'

2017/01/10 07:03:29 CLSRSC-473: Successfully exported the OCR

2017/01/10 07:03:33 CLSRSC-486:
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2017/01/10 07:03:33 CLSRSC-541:
 To downgrade the cluster:
 1. All nodes that have been upgraded must be downgraded.

2017/01/10 07:03:33 CLSRSC-542:
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2017/01/10 07:03:33 CLSRSC-543:
 3. The downgrade command must be run on the node bs01db04 with the '-lastnode' option to restore global configuration data.

2017/01/10 07:04:04 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2017/01/10 07:04:13 CLSRSC-474: Initiating upgrade of resource types

2017/01/10 07:04:38 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.4.0 -d 12.1.0.2.0 -p first'

2017/01/10 07:04:38 CLSRSC-475: Upgrade of resource types successfully initiated.

2017/01/10 07:04:43 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

So here is the problem.... How to resolve it?

Follow the steps below to resolve the issue and proceed with Database upgrade process.

Solution

The workaround is simple, all you have to do is modify the sqlnet.ora in the Grid Infrastructure home to either set sqlnet.ora parameter "DIAG_ADR_ENABLED" to ON (the default value), or set sqlnet.ora parameter "DIAG_ADR_ENABLED" to OFF and set  environment variable "ORA_CLIENTTRACE_DIR" to a valid directory.
dm01db01-+ASM1 {/home/oracle}:cd /u01/app/12.1.0.2/grid/network/admin/
 

dm01db01-+ASM1 {/u01/app/12.1.0.2/grid/network/admin}:ls -ltr sqlnet.ora
-rw-r--r-- 1 oracle oinstall 434 Jan 10 06:53 sqlnet.ora
 

dm01db01-+ASM1 {/u01/app/12.1.0.2/grid/network/admin}:vi sqlnet.ora
 

dm01db01-+ASM1 {/u01/app/12.1.0.2/grid/network/admin}:cat sqlnet.ora
# sqlnet.ora.dm01db01 Network Configuration File: /u01/app/11.2.0/grid/network/admin/sqlnet.ora.dm01db01
# Generated by Oracle configuration tools.

ADR_BASE = /u01/app/oracle
#SQLNET.AUTHENTICATION_SERVICES = (ALL)
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

SQLNET.DEFAULT_SDU_SIZE=32767

SQLNET.INBOUND_CONNECT_TIMEOUT=90
DIAG_ADR_ENABLED=ON
ORA_CLIENTTRACE_DIR=/u01/trace

TRACE_TIMESTAMP_CLIENT = ON
TRACE_DIRECTORY_SERVER = /u01/trace


Now verify the CRS active version

dm01db01-+ASM1 {/home/oracle}:dcli -g dbs_group -l oracle '/u01/app/12.1.0.2/grid/bin/crsctl query crs activeversion'
dm01db01: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db02: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db03: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db04: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db05: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db06: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db07: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db08: Oracle Clusterware active version on the cluster is [12.1.0.2.0]


Conclusion
It is possible that you GI and Database upgrade process fail due prerequisite failure or upgrade BUGS. In my case the rootupgrade.sh overall was successful but there were couple of lines that went unnoticed and caused the database upgrade problem. We have seen how to fix that issue and resolve the database upgrade issue.

No comments:

Post a Comment

Comparing Oracle Database Appliance X8-2 Model Family

September 2019 Oracle announced Oracle Database Appliance X8-2 (Small, Medium and HA). ODA X8-2 comes with more computing resources com...