Exadata: Upgrade Oracle Grid Infrastructure and Database from 11.2.0.4 to 12.1.0.2 - Part 1

Overview
It's time to upgrade Oracle Database 11g (11.2.0.4 in this article) to Oracle Database 12.1.0.2. In this article I will demostrate the Step by Step how to upgrade Oracle Grid/Database 11.2.0.4 to 12.1.0.2 on Exadata Database Machine X5-2 Full Rack.

Since the upgrade task is very lengthy, I will split the article in 2 different parts:

• Part I:

• Prepare the Current Exadata Environment &
• Install and Upgrade Grid Infrastructure to 12.1.0.2

• Part II:

• Install Database 12.1.0.2 Software &
• Upgrade Database to 12.1.0.2 using DBCA and Post Upgrade Steps

Current Environment Details:

• Exadata Database Machine X5-2 Full Rack
• Exadata Storage Software version 12.1.2.1.1
• Oracle Grid/Database Version 11.2.0.4

Download Grid/Database Software and Copy it Exadata Compute node 1:

Follow the steps by step instruction below to download the Grid/Database software for upgrade

dm01db01-orcldb1 {/u01/app/oracle/software}:ls -l
-rw-r--r-- 1 oracle oinstall 1673519571 Jan  9 00:30 p21419221_121020_Linux-x86-64_1of10.zip
-rw-r--r-- 1 oracle oinstall 1014527110 Jan  9 00:30 p21419221_121020_Linux-x86-64_2of10.zip
-rw-r--r-- 1 oracle oinstall  646969279 Jan  9 01:25 p21419221_121020_Linux-x86-64_6of10.zip
-rw-r--r-- 1 oracle oinstall 1747021273 Jan  9 01:38 p21419221_121020_Linux-x86-64_5of10.zip

Note: There are 2 zip files each for Database and Grid Infrastructure.

• zip files 1of10 and 2of10 are Database specific files
• zip files 5of10 and 6of10 are Grid specific files

Unzip of both zip files in same directory as follows:

• Unzip database zip files

dm01db01-orcldb1 {/u01/app/oracle/software}:unzip p21419221_121020_Linux-x86-64_1of10.zip
Archive:  p21419221_121020_Linux-x86-64_1of10.zip
   creating: database/
  inflating: database/runInstaller
   creating: database/rpm/
  inflating: database/rpm/cvuqdisk-1.0.9-1.rpm
   creating: database/install/
  inflating: database/install/attachHome.sh
  inflating: database/install/oraparam.ini.deinstall
   creating: database/install/images/
  inflating: database/install/images/billboards.gif
   creating: database/install/resource/
  inflating: database/install/resource/cons.nls
...
  inflating: database/stage/Actions/launchPadActions/10.1.0.2.0/1/launchpadaction.jar
   creating: database/response/
  inflating: database/response/netca.rsp
  inflating: database/response/dbca.rsp
  inflating: database/response/db_install.rsp
  inflating: PatchSearch.xml

dm01db01-orcldb1 {/u01/app/oracle/software}:unzip p21419221_121020_Linux-x86-64_2of10.zip
Archive:  p21419221_121020_Linux-x86-64_2of10.zip
   creating: database/stage/Components/oracle.ctx/
   creating: database/stage/Components/oracle.ctx/12.1.0.2.0/
   creating: database/stage/Components/oracle.ctx/12.1.0.2.0/1/
   creating: database/stage/Components/oracle.ctx/12.1.0.2.0/1/DataFiles/
  inflating: database/stage/Components/oracle.ctx/12.1.0.2.0/1/DataFiles/filegroup15.21.1.jar
  inflating: database/stage/Components/oracle.ctx/12.1.0.2.0/1/DataFiles/filegroup15.9.1.jar
  inflating: database/stage/Components/oracle.ctx/12.1.0.2.0/1/DataFiles/filegroup3.jar
  inflating: database/stage/Components/oracle.ctx/12.1.0.2.0/1/DataFiles/filegroup15.16.1.jar
  inflating: database/stage/Components/oracle.ctx/12.1.0.2.0/1/DataFiles/filegroup4.jar
...
  inflating: database/stage/Components/oracle.rdbms/12.1.0.2.0/1/DataFiles/filegroup40.jar
   creating: database/stage/Components/oracle.javavm.containers/
   creating: database/stage/Components/oracle.javavm.containers/12.1.0.2.0/
   creating: database/stage/Components/oracle.javavm.containers/12.1.0.2.0/1/
   creating: database/stage/Components/oracle.javavm.containers/12.1.0.2.0/1/DataFiles/
  inflating: database/stage/Components/oracle.javavm.containers/12.1.0.2.0/1/DataFiles/filegroup1.jar
  inflating: database/stage/Components/oracle.javavm.containers/12.1.0.2.0/1/DataFiles/filegroup2.jar
  inflating: database/install/.oui

• Unzip Grid Infrastructure zip files

dm01db01-orcldb1 {/u01/app/oracle/software}:unzip p21419221_121020_Linux-x86-64_5of10.zip
Archive:  p21419221_121020_Linux-x86-64_5of10.zip
   creating: grid/
  inflating: grid/runcluvfy.sh
  inflating: grid/welcome.html
   creating: grid/install/
  inflating: grid/install/oraparam.ini
  inflating: grid/install/clusterparam.ini
  inflating: grid/install/detachHome.sh
  inflating: grid/install/cvu.ini
  inflating: grid/install/lsnodes
   creating: grid/install/resource/
  inflating: grid/install/resource/cons_pt_BR.nls
  inflating: grid/install/resource/cons_es.nls
  inflating: grid/install/resource/cons_it.nls
  inflating: grid/install/resource/cons.nls
  inflating: grid/install/resource/cons_ko.nls
  inflating: grid/install/resource/cons_zh_TW.nls
...
 extracting: grid/stage/sizes/oracle.crs.Complete.sizes.properties
  inflating: grid/stage/sizes/oracle.crs12.1.0.2.0Complete.sizes.properties
   creating: grid/response/
  inflating: grid/response/grid_install.rsp
   creating: grid/sshsetup/
  inflating: grid/sshsetup/sshUserSetup.sh
  inflating: grid/runInstaller
  inflating: grid/readme.html
   creating: grid/rpm/
  inflating: grid/rpm/cvuqdisk-1.0.9-1.rpm

dm01db01-orcldb1 {/u01/app/oracle/software}:unzip p21419221_121020_Linux-x86-64_6of10.zip
Archive:  p21419221_121020_Linux-x86-64_6of10.zip
   creating: grid/stage/Components/oracle.has.crs/
   creating: grid/stage/Components/oracle.has.crs/12.1.0.2.0/
   creating: grid/stage/Components/oracle.has.crs/12.1.0.2.0/1/
   creating: grid/stage/Components/oracle.has.crs/12.1.0.2.0/1/DataFiles/
  inflating: grid/stage/Components/oracle.has.crs/12.1.0.2.0/1/DataFiles/filegroup44.jar
  inflating: grid/stage/Components/oracle.has.crs/12.1.0.2.0/1/DataFiles/filegroup8.jar
  inflating: grid/stage/Components/oracle.has.crs/12.1.0.2.0/1/DataFiles/filegroup38.jar
  inflating: grid/stage/Components/oracle.has.crs/12.1.0.2.0/1/DataFiles/filegroup43.jar
  inflating: grid/stage/Components/oracle.has.crs/12.1.0.2.0/1/DataFiles/filegroup31.jar
...
  inflating: grid/stage/Components/oracle.rdbms/12.1.0.2.0/1/DataFiles/filegroup19.4.1.jar
  inflating: grid/stage/Components/oracle.rdbms/12.1.0.2.0/1/DataFiles/filegroup73.jar
  inflating: grid/stage/Components/oracle.rdbms/12.1.0.2.0/1/DataFiles/filegroup19.8.1.jar
  inflating: grid/stage/Components/oracle.rdbms/12.1.0.2.0/1/DataFiles/filegroup4.jar
  inflating: grid/stage/Components/oracle.rdbms/12.1.0.2.0/1/DataFiles/filegroup3.jar
  inflating: grid/stage/Components/oracle.rdbms/12.1.0.2.0/1/DataFiles/filegroup19.22.1.jar
  inflating: grid/stage/Components/oracle.rdbms/12.1.0.2.0/1/DataFiles/filegroup74.jar
  inflating: grid/install/.oui

dm01db01-orcldb1 {/u01/app/oracle/software}:ls -ltr
total 4967820
drwxr-xr-x 7 oracle oinstall       4096 Jul 11  2014 database
drwxr-xr-x 7 oracle oinstall       4096 Jul 11  2014 grid
-rw-rw-r-- 1 oracle oinstall       6584 Aug 13  2015 PatchSearch.xml
-rw-r--r-- 1 oracle oinstall 1673519571 Jan  9 00:30 p21419221_121020_Linux-x86-64_1of10.zip
-rw-r--r-- 1 oracle oinstall 1014527110 Jan  9 00:30 p21419221_121020_Linux-x86-64_2of10.zip
-rw-r--r-- 1 oracle oinstall  646969279 Jan  9 01:25 p21419221_121020_Linux-x86-64_6of10.zip
-rw-r--r-- 1 oracle oinstall 1747021273 Jan  9 01:38 p21419221_121020_Linux-x86-64_5of10.zip

Things to consider before upgrade to Oracle Grid Infrastructure 12c and Oracle Database 12c:

• Current Oracle Database and Grid Infrastructure version must be 11.2.0.2, 11.2.0.3, 11.2.0.4 or 12.1.0.1.

dm01db01-+ASM1 {/home/oracle}:sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Mon Jan 9 00:33:25 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

dm01db01-orcldb1 {/home/oracle}:sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Jan 9 00:32:24 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

• Oracle Database 12c and Exadata

For full Exadata functionality including:

• Smart Scan offloaded filtering
• storage indexes
• I/O Resource Management (IORM),

Exadata Storage Server version 12.1.1.1.0 is required and 12.1.1.1.1 is recommended for Oracle Databases Release 12.1.0.2.

[root@dm01db01 ~]# imageinfo
Kernel version: 2.6.39-400.248.3.el6uek.x86_64 #1 SMP Wed Mar 11 18:04:34 PDT 2015 x86_64
Image version: 12.1.2.1.1.150316.2
Image activated: 2015-04-11 15:17:40 -0500
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1

Note: Customers unable to upgrade to Exadata 12.1.1.1.1 require a minimum version of 11.2.3.3.1 or later on Exadata Storage Servers and Database Servers. See document 1537407.1 for restrictions when running Oracle Database 12c in combination with Exadata releases earlier than 12.1.1.1.0.

• Sun Datacenter InfiniBand Switch 36 is running software release 1.3.3-2 or later, 2.1.3-4 recommended

[root@dm01db01 ~]# ssh dm01sw-ib1 version
root@dm01sw-ib1's password:
SUN DCS 36p version: 2.1.5-1
Build time: Oct  6 2014 10:35:15
SP board info:
Manufacturing Date: 2012.09.12
Serial Number: "NCDA70501"
Hardware Revision: 0x0007
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

In my case I am running ESS version 12.1.2.1.1 and IB Switch software version 2.1.5-1.

• Fix for bug 12539000 is required to successfully upgrade.

11.2.0.2 BP12 and later, 11.2.0.3, 11.2.0.4 and 12.1.0.1 already contain this fix. An interim patch must be installed for 11.2.0.2 Grid Infrastructure and Database installations running BP11 or earlier.

• Fix for bug 14639430 is required to properly rollback Grid Infrastructure upgrade, if necessary (not required when Grid Infrastructure already is on 11.2.0.4 or 12.1.0.1).

• Fix for bug 13460353 is required to create a new 11g database after Grid Infrastructure is upgraded to 12c.(not required when database already is on 11.2.0.4 or 12.1.0.1).

• When available: GI PSU 12.1.0.2.1 or later (which includes DB PSU 12.1.0.2.1). To be applied:

•     during the upgrade process, before running rootupgrade.sh on the Grid Infrastructure home, or

• after installing the new Database home, before upgrading the database.

• Grid Infrastructure upgrades on nodes with different length hostnames in the same cluster require fix for bug 19453778 - CTSSD FAILED TO START WHILE RUNNING ROOTUPGRADE.SH. Contact Oracle Support to obtain the patch.

In my case I am upgrading from 11.2.0.4 to 12.1.0.2, these bugs are already fixed. So I don't need to apply any of these Bug fixes.

High level steps involved in Oracle Grid/Database Upgrade process

• Pre-Upgrade steps: Preparing the current environment
• Install and Upgrade Oracle Grid Infrastructure to 12.1.0.2
• Install Oracle Database 12.1.0.2 Software
• Upgrade Oracle Database(s) to 12.1.0.2
• Post-upgrade steps
• Troubleshooting steps

Assumptions

• The Oracle Grid/Databasse software owner is oracle.

dm01db01-orcldb1 {/u01/app/oracle/software}:id oracle
uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)

• The Oracle inventory group is oinstall.

dm01db01-orcldb1 {/u01/app/oracle/software}:ls -ld /u01/app/oraInventory/
drwxrwx--- 8 oracle oinstall 4096 Jan  8 04:44 /u01/app/oraInventory/

• Create dbs_group file in Oracle and Root users home directory and specify the all compute nodes one per line.

dm01db01-orcldb1 {/u01/app/oracle/software}:cat ~/dbs_group
dm01db01
dm01db02
dm01db03
dm01db04
dm01db05
dm01db06
dm01db07
dm01db08

• Current database home. In my case database version is 11.2.0.4

dm01db01-orcldb1 {/u01/app/oracle/software}:echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0.4/dbhome

• Current Grid Infrastructure home. In my case grid version is 11.2.0.4

dm01db01-+ASM1 {/u01/app/oracle/software}:echo $ORACLE_HOME
/u01/app/11.2.0.4/grid

• The primary database to be upgraded is named "orcldb".

• Run Exadata Health Check (Exachk) and make sure no major hardware/software issues are reported.

Important MOS note:
Read them carefully before starting upgrade process:

• Document 888828.1 - Database Machine and Exadata Storage Server Supported Releases
• Document 1537407.1 - Requirements and restrictions when using Oracle Database 12c on Exadata Database Machine
• Document 1270094.1 - Exadata Critical Issues
• Document 1070954.1 - Oracle Exadata Database Machine exachk or HealthCheck

Pre-Upgrade steps: Preparing the current environment

• Test the Upgrade on non-production environment first

• Ensure Oracle Grid/Database Homes are backed up before upgrade

As root user:
 

Database Home backup:
[oracle@pr04db01 ~]# cd /u01/app/oracle/product/11.2.0.4/dbhome_1
[oracle@pr04db01 ~]# tar –zcvf /u01/app/oracle/product/11.2.0.4/db11204.tgz .

Grid Home backup:
[oracle@pr04db01 ~]# cd /u01/app/11.2.0.4/grid
[oracle@pr04db01 ~]# tar –zcvf /u01/app/11.2.0.4/gi11204.tgz .

• Optionally take a snapshot backup of /u01 file system

• Ensure the databases are backup before upgrade. Using RMAN backup to FRA or ZFS is recommended.

RMAN> backup database plus archivelog;

• It is recommended to create Guaranteed Restore Point in addition to Database backup

SQL> CREATE RESTORE POINT before_upgrade_12102 GUARANTEE FLASHBACK DATABASE;

• Perform one-off patch assesment

Verify that one-off patches currently installed on top of 11.2.0.4 are fixed in 12.1.0.2. Review the README. Contact Oracle Support if you are unable to determine if a one-off patch is still required on top of 12.1.0.2.

• Do not place the new ORACLE_HOME under /opt/oracle

• Download 12.1.0.2 Grid/Database software - 21419221

• Download latest Opatch utility software - 6880880

• Download one-off patch required for your existing software version if any.

• Apply patches where required before upgrading proceeds

• Run Exachk before upgrade

Validate Readiness for Oracle Clusterware upgrade using CVU and Exachck

• Navigate to grid software directory and run the Cluster verification utility as follows:

dm01db01-+ASM1 {/u01/app/oracle/software}:cd grid/
dm01db01-+ASM1 {/u01/app/oracle/software/grid}:ls -ltr
total 80
-rwxr-xr-x  1 oracle oinstall   500 Feb  6  2013 welcome.html
-rwxr-xr-x  1 oracle oinstall  5085 Dec 20  2013 runcluvfy.sh
-rwxr-xr-x  1 oracle oinstall  8534 Jul  7  2014 runInstaller
drwxr-xr-x  2 oracle oinstall  4096 Jul  7  2014 rpm
drwxrwxr-x  2 oracle oinstall  4096 Jul  7  2014 sshsetup
drwxrwxr-x  2 oracle oinstall  4096 Jul  7  2014 response
drwxr-xr-x 14 oracle oinstall  4096 Jul  7  2014 stage
-rwxr-xr-x  1 oracle oinstall 33934 Aug  7  2015 readme.html
drwxr-xr-x  4 oracle oinstall  4096 Jan  9 04:43 install

dm01db01-+ASM1 {/u01/app/oracle/software/grid}:./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/11.2.0.4/grid -dest_crshome /u01/app/12.1.0.2/grid -dest_version 12.1.0.2.0 -fixup -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "dm01db01"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  dm01db01                              yes
  dm01db02                              yes
  dm01db03                              yes
  dm01db04                              yes
  dm01db05                              yes
  dm01db06                              yes
  dm01db07                              yes
  dm01db08                              yes
Result: Node reachability check passed from node "dm01db01"

Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  dm01db08                              passed
  dm01db07                              passed
  dm01db06                              passed
  dm01db05                              passed
  dm01db04                              passed
  dm01db03                              passed
  dm01db02                              passed
  dm01db01                              passed
Result: User equivalence check passed for user "oracle"

Checking CRS user consistency
Result: CRS user consistency check successful
Checking network configuration consistency.
Result: Check for network configuration consistency passed.
Checking ASM disk size consistency
All ASM disks are correctly sized
Checking if ASM parameter file is in use by an ASM instance on the local node
Result: ASM instance is using parameter file "+SYSTEMDG/dm01-cluster/asmparameterfile/registry.253.726236669" on

node "dm01db01" on which upgrade is requested.

Checking OLR integrity...
Check of existence of OLR configuration file "/etc/oracle/olr.loc" passed
Check of attributes of OLR configuration file "/etc/oracle/olr.loc" passed

WARNING:
This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local' as a privileged user to verify the contents of OLR.

OLR integrity check passed

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  dm01db01                              passed
  dm01db08                              passed
  dm01db07                              passed
  dm01db06                              passed
  dm01db05                              passed
  dm01db04                              passed
  dm01db03                              passed
  dm01db02                              passed

Verification of the hosts config file successful

................

Check for Reverse path filter setting passed

Starting check for Network interface bonding status of private interconnect network interfaces ...

Check for Network interface bonding status of private interconnect network interfaces passed

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Starting check for /boot mount ...

Check for /boot mount passed

Starting check for zeroconf check ...

Check for zeroconf check passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

NOTE:
No fixable verification failures to fix
 

- OR - 

You can also redirect the output to the file and read it easily.

$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/11.2.0.4/grid -dest_crshome /u01/app/12.1.0.2/grid -dest_version 12.1.0.2.0 -fixup -verbose > cluvfy_pre_upgrade.log

Pre-upgrade Information Tool

• Analyze your databases to be upgraded with the Pre-Upgrade Information Tool. Download and copy Pre-upgradeutility. Take a look at the MOS note below:

How to Download and Run Oracle's Database Pre-Upgrade Utility (Doc ID 884522.1)

• Run the pre-upgrade utitlity as follows:

dm01db01-+ASM1 {/u01/app/oracle/software}:ls -ltr preupgrade_12.1.0.2.0_14_lf.zip
-rw-r--r-- 1 oracle oinstall 101162 Jan  9 06:19 preupgrade_12.1.0.2.0_14_lf.zip

dm01db01-+ASM1 {/u01/app/oracle/software}:unzip preupgrade_12.1.0.2.0_14_lf.zip
Archive:  preupgrade_12.1.0.2.0_14_lf.zip
  inflating: preupgrd.sql
  inflating: utluppkg.sql

dm01db01-+ASM1 {/u01/app/oracle/software}:. oraenv
ORACLE_SID = [+ASM1] ? orcldb1
The Oracle base remains unchanged with value /u01/app/oracle
dm01db01-orcldb1 {/u01/app/oracle/software}:sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Jan 9 06:25:51 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @preupgrd.sql

Loading Pre-Upgrade Package...

***************************************************************************
Executing Pre-Upgrade Checks in ORCLDB...
************************************************************

***************   ************************************************************
                  ====>> ERRORS FOUND for ORCLDB <<====
 The following are *** ERROR LEVEL CONDITIONS *** that must be addressed
                    prior to attempting your upgrade.
            Failure to do so will result in a failed upgrade.

           You MUST resolve the above errors prior to upgrade

     ************************************************************
      ************************************************************
              ====>> PRE-UPGRADE RESULTS for ORCLDB <<====

ACTIONS REQUIRED:

1. Review results of the pre-upgrade checks:
 /u01/app/oracle/cfgtoollogs/orcldb/preupgrade/preupgrade.log

2. Execute in the SOURCE environment BEFORE upgrade:
 /u01/app/oracle/cfgtoollogs/orcldb/preupgrade/preupgrade_fixups.sql

3. Execute in the NEW environment AFTER upgrade:
 /u01/app/oracle/cfgtoollogs/orcldb/preupgrade/postupgrade_fixups.sql

    ************************************************************

***************************************************************************
Pre-Upgrade Checks in ORCLDB Completed.
***************************************************************************

***************************************************************************
***************************************************************************

• Review the preupgrade.log file for errors/issues:

dm01db01-orcldb1 {/u01/app/oracle/software}:vi /u01/app/oracle/cfgtoollogs/orcldb/preupgrade/preupgrade.log

• Run the preupgrade_fixups.sql produced by pre-upgrade utility above:

dm01db01-orcldb1 {/u01/app/oracle/software}:sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Jan 9 06:29:50 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @/u01/app/oracle/cfgtoollogs/orcldb/preupgrade/preupgrade_fixups.sql
Pre-Upgrade Fixup Script Generated on 2017-01-09 06:26:48  Version: 12.1.0.2 Build: 014
Beginning Pre-Upgrade Fixups...
Executing in container ORCLDB

**********************************************************************
Check Tag:     DEFAULT_PROCESS_COUNT
Check Summary: Verify min process count is not too low
Fix Summary:   Review and increase if needed, your PROCESSES value.
**********************************************************************
Fixup Returned Information:
WARNING: --> Process Count may be too low

     Database has a maximum process count of 150 which is lower than the
     default value of 300 for this release.
     You should update your processes value prior to the upgrade
     to a value of at least 300.
     For example:
        ALTER SYSTEM SET PROCESSES=300 SCOPE=SPFILE
     or update your init.ora file.
**********************************************************************

**********************************************************************
Check Tag:     EM_PRESENT
Check Summary: Check if Enterprise Manager is present
Fix Summary:   Execute emremove.sql prior to upgrade.
**********************************************************************
Fixup Returned Information:
WARNING: --> Enterprise Manager Database Control repository found in the database

     In Oracle Database 12c, Database Control is removed during
     the upgrade. To save time during the Upgrade, this action
     can be done prior to upgrading using the following steps after
     copying rdbms/admin/emremove.sql from the new Oracle home
   - Stop EM Database Control:
    $> emctl stop dbconsole

   - Connect to the Database using the SYS account AS SYSDBA:

   SET ECHO ON;
   SET SERVEROUTPUT ON;
   @emremove.sql
     Without the set echo and serveroutput commands you will not
     be able to follow the progress of the script.
**********************************************************************

**********************************************************************
Check Tag:     AMD_EXISTS
Check Summary: Check to see if AMD is present in the database
Fix Summary:   Manually execute ORACLE_HOME/oraolap/admin/catnoamd.sql script to remove OLAP.
**********************************************************************
Fixup Returned Information:
INFORMATION: --> OLAP Catalog(AMD) exists in database

     Starting with Oracle Database 12c, OLAP Catalog component is desupported.
     If you are not using the OLAP Catalog component and want
     to remove it, then execute the
     ORACLE_HOME/olap/admin/catnoamd.sql script before or
     after the upgrade.
**********************************************************************

**********************************************************************
Check Tag:     APEX_UPGRADE_MSG
Check Summary: Check that APEX will need to be upgraded.
Fix Summary:   Oracle Application Express can be manually upgraded prior to database upgrade.
**********************************************************************
Fixup Returned Information:
INFORMATION: --> Oracle Application Express (APEX) can be
     manually upgraded prior to database upgrade

     APEX is currently at version 3.2.1.00.12 and will need to be
     upgraded to APEX version 4.2.5 in the new release.
     Note 1: To reduce database upgrade time, APEX can be manually
             upgraded outside of and prior to database upgrade.
     Note 2: See MOS Note 1088970.1 for information on APEX
             installation upgrades.
**********************************************************************

**********************************************************************
                      [Pre-Upgrade Recommendations]
**********************************************************************

                        *****************************************
                        ********* Dictionary Statistics *********
                        *****************************************

Please gather dictionary statistics 24 hours prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:
    EXECUTE dbms_stats.gather_dictionary_stats;

^^^ MANUAL ACTION SUGGESTED ^^^

           **************************************************
                ************* Fixup Summary ************

 4 fixup routines generated INFORMATIONAL messages that should be reviewed.

**************** Pre-Upgrade Fixup Script Complete *********************

PL/SQL procedure successfully completed.

• Take care of pre-upgrade Recommendation stated above:
• Increase processes parameter to 300

SQL> ALTER SYSTEM SET PROCESSES=300 SCOPE=SPFILE;

System altered.

• Drop OLAP Catalog

SQL> @$ORACLE_HOME/olap/admin/catnoamd.sql

Synonym dropped.

Synonym dropped.

Synonym dropped.

Synonym dropped.

Synonym dropped.

Synonym dropped.

....

Type dropped.

Type dropped.

PL/SQL procedure successfully completed.

Role dropped.

PL/SQL procedure successfully completed.

1 row deleted.

• Gather dictionary statistics

SQL> EXECUTE dbms_stats.gather_dictionary_stats;

PL/SQL procedure successfully completed.

Install and Upgrade Grid Infrastructure to 12.1.0.2

• Create the new Grid Infrastructure (GI_HOME) directory where 12.1.0.2 will be installed

[root@dm01db01 ~]# dcli -g ~/dbs_group -l root mkdir -p /u01/app/12.1.0.2/grid/
[root@dm01db01 ~]# dcli -g ~/dbs_group -l root chown oracle /u01/app/12.1.0.2/grid
[root@dm01db01 ~]# dcli -g ~/dbs_group -l root chgrp -R oinstall /u01/app/12.1.0.2/grid
[root@dm01db01 ~]# dcli -g ~/dbs_group -l root ls -l  /u01/app/12.1.0.2/
dm01db01: total 4
dm01db01: drwxr-xr-x 2 oracle oinstall 4096 Jan 10 03:44 grid
dm01db02: total 4
dm01db02: drwxr-xr-x 2 oracle oinstall 4096 Jan 10 03:44 grid
dm01db03: total 4
dm01db03: drwxr-xr-x 2 oracle oinstall 4096 Jan 10 03:44 grid
dm01db04: total 4
dm01db04: drwxr-xr-x 2 oracle oinstall 4096 Jan 10 03:44 grid
dm01db05: total 4
dm01db05: drwxr-xr-x 2 oracle oinstall 4096 Jan 10 03:44 grid
dm01db06: total 4
dm01db06: drwxr-xr-x 2 oracle oinstall 4096 Jan 10 03:44 grid
dm01db07: total 4
dm01db07: drwxr-xr-x 2 oracle oinstall 4096 Jan 10 03:44 grid
dm01db08: total 4
dm01db08: drwxr-xr-x 2 oracle oinstall 4096 Jan 10 03:44 grid

• Unzip the software if not already done earlier.

dm01db01-orcldb1 {/u01/app/oracle/software}: unzip -q /u01/app/oracle/software/p21419221_121020_Linux-x86-64_5of10.zip -d /u01/app/oracle/software

dm01db01-orcldb1 {/u01/app/oracle/software}: unzip -q /u01/app/oracle/software/p21419221_121020_Linux-x86-64_6of10.zip -d /u01/app/oracle/software

• Unset the environment and run the installer

dm01db01-orcldb1 {/u01/app/oracle/software}:unset ORACLE_HOME ORACLE_BASE ORACLE_SID
dm01db01- {/u01/app/oracle/software}:echo $ORACLE_HOME

dm01db01- {/u01/app/oracle/software}:echo $ORACLE_BASE

dm01db01- {/u01/app/oracle/software}:echo $ORACLE_SID

dm01db01- {/home/oracle}:export SRVM_USE_RACTRANS=true
dm01db01- {/home/oracle}:export DISPLAY=10.10.20.1:0.0
dm01db01- {/home/oracle}:cd /u01/app/oracle/software/grid/
dm01db01- {/u01/app/oracle/software/grid}:./runInstaller -J-Doracle.install.mgmtDB=false
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB.   Actual 7663 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 16378 MB    Passed
Checking monitor: must be configured to display at least 256 colors
   
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2017-01-10_03-50-56AM. Please wait ...dm01db01- {/u01/app/oracle/software/grid}:

Select "Upgrade Oracle GI or ASM" and click Next

Select Language and click Next

Select all the nodes you want to upgrade and click Next

Click Next

Select the OS group for OS authentication and click Next

Enter the Oracle Base and Grid Home and click Next

Click Next as we want to run the root script manually.

Prerequisites check in progress

Click Install to begin Installation process

 Installation in progress

Follow the steps below before executing rootupgrade.sh script

• Update OPatch and when available apply the latest Bundle Patch on top of the 12.1.0.2 Grid Infrastructure Installation

dm01db01-orcldb1 {/u01/app/oracle/software}:dcli -g ~/dbs_group -l oracle -d /u01/app/oracle/software -f p6880880_112000_Linux-x86-64.zip

dm01db01-orcldb1 {/u01/app/oracle/software}:dcli -g ~/dbs_group -l oracle unzip -oq -d /u01/app/12.1.0.2/grid  /u01/app/oracle/software/p6880880_112000_Linux-x86-64.zip

dm01db01-orcldb1 {/u01/app/oracle/software}:dcli -g ~/dbs_group -l oracle /u01/app/12.1.0.2/grid/OPatch/opatch version
dm01db01: OPatch Version: 11.2.0.3.15
dm01db01:
dm01db01: OPatch succeeded.
dm01db02: OPatch Version: 11.2.0.3.15
dm01db02:
dm01db02: OPatch succeeded.
dm01db03: OPatch Version: 11.2.0.3.15
dm01db03:
dm01db03: OPatch succeeded.
dm01db04: OPatch Version: 11.2.0.3.15
dm01db04:
dm01db04: OPatch succeeded.
dm01db05: OPatch Version: 11.2.0.3.15
dm01db05:
dm01db05: OPatch succeeded.
dm01db06: OPatch Version: 11.2.0.3.15
dm01db06:
dm01db06: OPatch succeeded.
dm01db07: OPatch Version: 11.2.0.3.15
dm01db07:
dm01db07: OPatch succeeded.
dm01db08: OPatch Version: 11.2.0.3.15
dm01db08:
dm01db08: OPatch succeeded.

• When required relink the 12.1.0.2 Grid Infrastructure oracle binary with RDS

dm01db01-orcldb1 {/u01/app/oracle/software}:dcli -g ~/dbs_group -l oracle /u01/app/12.1.0.2/grid/bin/skgxpinfo
dm01db01: rds
dm01db02: rds
dm01db03: rds
dm01db04: rds
dm01db05: rds
dm01db06: rds
dm01db07: rds
dm01db08: rds

If the command does not return 'rds' relink as follows:
 

dm01db01-orcldb1 {/u01/app/oracle/software}: dcli -g ~/dbs_group -l oracle ORACLE_HOME=/u01/app/12.1.0.2/grid make -C /u01/app/12.1.0.2/grid/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle

• Change SGA memory settings for ASM

dm01db01-+ASM1 {/u01/app/oracle/software}:sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Tue Jan 10 06:35:32 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter system set sga_max_size = 2G scope=spfile sid='*';

System altered.

SQL> alter system set sga_target = 2G scope=spfile sid='*';

System altered.

• Verify values for memory_target, memory_max_target and use_large_pages

SQL> col sid format a4
SQL> col name format a25
SQL> col value format a20
SQL> set lines 200
SQL> set pages 200
SQL> select sid, name, value from v$spparameter where name in('memory_target','memory_max_target','use_large_pages');

SID   NAME                           VALUE
---- -------------------------          --------------------
*     use_large_pages                TRUE
*     memory_target                  0
*     memory_max_target

• Verify no active rebalance is running

SQL> select count(*) from gv$asm_operation;

  COUNT(*)
----------
         0

• Now Execute rootupgrade.sh on each database server

Note: Run rootupgrade.sh on node 1 first. After rootupgrade.sh completes successfully on the local node, you can run the script in parallel on other nodes except for the last node. When the script has completed successfully on all the nodes except the last node, run the script on the last node.  Do not run rootupgrade.sh on the last node until the script has run successfully on all other nodes.
 

On Node 1

[root@dm01db01 ~]# id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10
[root@dm01db01 ~]# /u01/app/12.1.0.2/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.2/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2017/01/10 06:52:50 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA)

Collector.

2017/01/10 06:52:51 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2017/01/10 06:52:53 CLSRSC-464: Starting retrieval of the cluster configuration data

2017/01/10 06:53:02 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2017/01/10 06:53:02 CLSRSC-363: User ignored prerequisites during installation

2017/01/10 06:53:13 CLSRSC-515: Starting OCR manual backup.

2017/01/10 06:53:15 CLSRSC-516: OCR manual backup successful.

2017/01/10 06:53:19 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2017/01/10 06:53:19 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/asmca -silent -upgradeNodeASM -

nonRolling false -oldCRSHome /u01/app/11.2.0.4/grid -oldCRSVersion 11.2.0.4.0 -nodeNumber 1 -firstNode true -startRolling true'

ASM configuration upgraded in local node successfully.

2017/01/10 06:53:27 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode

2017/01/10 06:53:27 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2017/01/10 06:53:53 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

OLR initialization - successful
2017/01/10 06:58:48 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
kgfnGetFacility: facility=0x1564768
kgfnInitDiag: diagctx=0x13e4160
CRS-4123: Oracle High Availability Services has been started.
2017/01/10 07:03:18 CLSRSC-472: Attempting to export the OCR

2017/01/10 07:03:18 CLSRSC-482: Running command: 'ocrconfig -upgrade oracle oinstall'

2017/01/10 07:03:29 CLSRSC-473: Successfully exported the OCR

2017/01/10 07:03:33 CLSRSC-486:
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2017/01/10 07:03:33 CLSRSC-541:
 To downgrade the cluster:
 1. All nodes that have been upgraded must be downgraded.

2017/01/10 07:03:33 CLSRSC-542:
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2017/01/10 07:03:33 CLSRSC-543:
 3. The downgrade command must be run on the node dm01db04 with the '-lastnode' option to restore global configuration data.

2017/01/10 07:04:04 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2017/01/10 07:04:13 CLSRSC-474: Initiating upgrade of resource types

2017/01/10 07:04:38 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.4.0 -d 12.1.0.2.0 -p first'

2017/01/10 07:04:38 CLSRSC-475: Upgrade of resource types successfully initiated.

2017/01/10 07:04:43 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Now execute the rootupgrade.sh on all the other nodes in the cluster.

Go back to the installation screen and click Ok

Click Close to complete the upgrade process.

Verify cluster status

dm01db01-+ASM1 {/home/oracle}:dcli -g dbs_group -l oracle '/u01/app/12.1.0.2/grid/bin/crsctl query crs activeversion'
dm01db01: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db02: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db03: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db04: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db05: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db06: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db07: Oracle Clusterware active version on the cluster is [12.1.0.2.0]
dm01db08: Oracle Clusterware active version on the cluster is [12.1.0.2.0]

dm01db01-+ASM1 {/home/oracle}:dcli -g dbs_group -l oracle '/u01/app/12.1.0.2/grid/bin/crsctl query crs softwareversion'
dm01db01: Oracle Clusterware version on node [dm01db01] is [12.1.0.2.0]
dm01db02: Oracle Clusterware version on node [dm01db02] is [12.1.0.2.0]
dm01db03: Oracle Clusterware version on node [dm01db03] is [12.1.0.2.0]
dm01db04: Oracle Clusterware version on node [dm01db04] is [12.1.0.2.0]
dm01db05: Oracle Clusterware version on node [dm01db05] is [12.1.0.2.0]
dm01db06: Oracle Clusterware version on node [dm01db06] is [12.1.0.2.0]
dm01db07: Oracle Clusterware version on node [dm01db07] is [12.1.0.2.0]
dm01db08: Oracle Clusterware version on node [dm01db08] is [12.1.0.2.0]

dm01db01-+ASM1 {/home/oracle}:/u01/app/12.1.0.2/grid/bin/crsctl check cluster -all
**************************************************************
dm01db01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dm01db02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dm01db03:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dm01db04:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dm01db05:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dm01db06:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dm01db07:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dm01db08:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
************************************************************** 

Change Custom Scripts and environment variables to Reference the 12.1.0.2 Grid Home

dm01db01-+ASM1 {/home/oracle}:vi .bash_profile
dm01db01-+ASM1 {/home/oracle}:vi .exadata_profile

SQL> set lines 200
SQL> set pages 200
SQL> select * from v$version;

BANNER                                                                               CON_ID
-------------------------------------------------------------------------------- ----------
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production              0
PL/SQL Release 12.1.0.2.0 - Production                                                    0
CORE    12.1.0.2.0      Production                                                                0
TNS for Linux: Version 12.1.0.2.0 - Production                                            0
NLSRTL Version 12.1.0.2.0 - Production                                                    0

ASMCMD> lsct -g DATA
Instance_ID  DB_Name  Status     Software_Version  Compatible_version  Instance_Name  Disk_Group
          1  +ASM     CONNECTED        12.1.0.2.0          12.1.0.2.0  +ASM1          DATA
          2  +ASM     CONNECTED        12.1.0.2.0          12.1.0.2.0  +ASM2          DATA
          3  +ASM     CONNECTED        12.1.0.2.0          12.1.0.2.0  +ASM3          DATA
          4  +ASM     CONNECTED        12.1.0.2.0          12.1.0.2.0  +ASM4          DATA
          5  +ASM     CONNECTED        12.1.0.2.0          12.1.0.2.0  +ASM5          DATA
          6  +ASM     CONNECTED        12.1.0.2.0          12.1.0.2.0  +ASM6          DATA
          7  +ASM     CONNECTED        12.1.0.2.0          12.1.0.2.0  +ASM7          DATA
          8  +ASM     CONNECTED        12.1.0.2.0          12.1.0.2.0  +ASM8          DATA
          1  orcldb   CONNECTED        11.2.0.4.0          11.2.0.4.0  orcldb1        DATA
          2  orcldb   CONNECTED        11.2.0.4.0          11.2.0.4.0  orcldb2        DATA
          3  orcldb   CONNECTED        11.2.0.4.0          11.2.0.4.0  orcldb3        DATA
          4  orcldb   CONNECTED        11.2.0.4.0          11.2.0.4.0  orcldb4        DATA
          5  orcldb   CONNECTED        11.2.0.4.0          11.2.0.4.0  orcldb5        DATA
          6  orcldb   CONNECTED        11.2.0.4.0          11.2.0.4.0  orcldb6        DATA
          7  orcldb   CONNECTED        11.2.0.4.0          11.2.0.4.0  orcldb7        DATA
          8  orcldb   CONNECTED        11.2.0.4.0          11.2.0.4.0  orcldb8        DATA

[root@dm01db01 ~]# /u01/app/12.1.0.2/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :      10592
         Available space (kbytes) :     398976
         ID                       : 1119121028
         Device/File Name         :  +SYSTEMDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded


dm01db01-+ASM1 {/home/oracle}:crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6938ea097e324f0fbf48da02892564cb (o/192.168.1.9/SYSTEMDG_CD_02_dm01cel01) [SYSTEMDG]
 2. ONLINE   42561e3fd68a4f7dbfe0687fb6bb6a1f (o/192.168.1.22/SYSTEMDG_CD_08_dm01cel14) [SYSTEMDG]
 3. ONLINE   2cc2afc24d504f99bfb05a60620e01ef (o/192.168.1.13/SYSTEMDG_CD_02_dm01cel05) [SYSTEMDG]
Located 3 voting disk(s).

Conclusion
In this article we have successfully upgraded Oracle Grid Infrastructure from 11.2.0.4 to 12.1.0.2. Oracle 12.1.0.2 includes the support for Smart Scan, Storage Indexes and IORM, so Customer can take full advantage of running 12.1.0.2 on Exadata Database Machine. The upgrade from 11.2.0.4 to 12.1.0.2 works seamless if all the prerequisites are met properly.