Steps to Shutdown Exadata Database Machine

Overview
When you perform power maintenance, data center maintenance or moving servers from one data center to another you need to shutdown the Clusterware/Database/Servers. If you have Exadata Database Machines sitting in your data center then you need to shutdown it as well. The process of shutting down Exadata Database machine is slightly different when compared to servers or cluster. We should follow a sequence of steps to shutdown Exadata components. 

In this article, we will demonstrate the steps to shutdown Exadata Database Machine to carry out maintenance task.

Environment
Exadata X5-2 Full Rack

• Log in to the first Exadata compute node as as root user.
• Change to root home directory and create the following files

[root@dm01db01 ~]# cd /root

[root@dm01db01 ~]# vi dbs_group
dm01db01
dm01db02
dm01db03
dm01db04
dm01db05
dm01db06
dm01db07
dm01db08

[root@dm01db01 ~]# vi cell_group
dm01cel01
dm01cel02
dm01cel03
dm01cel04
dm01cel05
dm01cel06
dm01cel07
dm01cel08
dm01cel09
dm01cel10
dm01cel11
dm01cel12
dm01cel13
dm01cel14

• Check if Grid Infrastructure is enabled for autostart:

[root@dm01db01 ~]# dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl config crs
dm01db01: CRS-4622: Oracle High Availability Services autostart is enabled.
dm01db02: CRS-4622: Oracle High Availability Services autostart is enabled.
dm01db03: CRS-4622: Oracle High Availability Services autostart is enabled.
dm01db04: CRS-4622: Oracle High Availability Services autostart is enabled.
dm01db05: CRS-4622: Oracle High Availability Services autostart is enabled.
dm01db06: CRS-4622: Oracle High Availability Services autostart is enabled.
dm01db07: CRS-4622: Oracle High Availability Services autostart is enabled.
dm01db08: CRS-4622: Oracle High Availability Services autostart is enabled.

• Disable the Grid Infrastructure for autostart on the compute nodes if it is currently enabled for autostart.

[root@dm01db01 ~]# dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl disable crs
dm01db01: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db02: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db03: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db04: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db05: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db06: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db07: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db08: CRS-4621: Oracle High Availability Services autostart is disabled.

[root@dm01db01 ~]# dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl config crs
dm01db01: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db02: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db03: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db04: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db05: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db06: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db07: CRS-4621: Oracle High Availability Services autostart is disabled.
dm01db08: CRS-4621: Oracle High Availability Services autostart is disabled.

• Shutdown the Grid Infrastructure stack on the Exadata compute nodes

This will take some time to come out
[root@dm01db01 ~]#dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl stop crs

[root@dm01db01 ~]# dcli -g dbs_group -l root /u01/app/11.2.0.4/grid/bin/crsctl check cluster
dm01db01: CRS-4639: Could not contact Oracle High Availability Services
dm01db01: CRS-4000: Command Check failed, or completed with errors.
dm01db02: CRS-4639: Could not contact Oracle High Availability Services
dm01db02: CRS-4000: Command Check failed, or completed with errors.
dm01db03: CRS-4639: Could not contact Oracle High Availability Services
dm01db03: CRS-4000: Command Check failed, or completed with errors.
dm01db04: CRS-4639: Could not contact Oracle High Availability Services
dm01db04: CRS-4000: Command Check failed, or completed with errors.
dm01db05: CRS-4639: Could not contact Oracle High Availability Services
dm01db05: CRS-4000: Command Check failed, or completed with errors.
dm01db06: CRS-4639: Could not contact Oracle High Availability Services
dm01db06: CRS-4000: Command Check failed, or completed with errors.
dm01db07: CRS-4639: Could not contact Oracle High Availability Services
dm01db07: CRS-4000: Command Check failed, or completed with errors.
dm01db08: CRS-4639: Could not contact Oracle High Availability Services
dm01db08: CRS-4000: Command Check failed, or completed with errors.

• Ensure that the Grid Infrastructure stack has shutdown successfully on all the Compute nodes. 

The following command returns no output if the Grid Infrastructure is shutdown:

[root@dm01db01 ~]# dcli -g dbs_group -l root "ps -ef | grep diskmo[n]"

• Disable email/SNMP alerts. The following command will show the current notification method:

[root@dm01db01 ~]# dcli -g cell_group -l root "cellcli -e list cell attributes name,notificationMethod"
dm01cel01: dm01cel01     none
dm01cel02: dm01cel02     none
dm01cel03: dm01cel03     none
dm01cel04: dm01cel04     none
dm01cel05: dm01cel05     none
dm01cel06: dm01cel06     none
dm01cel07: dm01cel07     none
dm01cel08: dm01cel08     none
dm01cel09: dm01cel09     none
dm01cel10: dm01cel10     none
dm01cel11: dm01cel11     none
dm01cel12: dm01cel12     none
dm01cel13: dm01cel13     none
dm01cel14: dm01cel14     none

• Now set the notification method to null to disable alerts:

[root@dm01db01 ~]# dcli -g cell_group -l root "cellcli -e alter cell notificationMethod=none"
dm01cel01: Cell dm01cel01 successfully altered
dm01cel02: Cell dm01cel02 successfully altered
dm01cel03: Cell dm01cel03 successfully altered
dm01cel04: Cell dm01cel04 successfully altered
dm01cel05: Cell dm01cel05 successfully altered
dm01cel06: Cell dm01cel06 successfully altered
dm01cel07: Cell dm01cel07 successfully altered
dm01cel08: Cell dm01cel08 successfully altered
dm01cel09: Cell dm01cel09 successfully altered
dm01cel10: Cell dm01cel10 successfully altered
dm01cel11: Cell dm01cel11 successfully altered
dm01cel12: Cell dm01cel12 successfully altered
dm01cel13: Cell dm01cel13 successfully altered
dm01cel14: Cell dm01cel14 successfully altered

• Shutdown cell services on all the cells:

[root@dm01db01 ~]# dcli -g cell_group -l root "cellcli -e alter cell shutdown services all"
dm01cel01:
dm01cel01: Stopping the RS, CELLSRV, and MS services...
dm01cel01: The SHUTDOWN of services was successful.
dm01cel02:
dm01cel02: Stopping the RS, CELLSRV, and MS services...
dm01cel02: The SHUTDOWN of services was successful.
dm01cel03:
dm01cel03: Stopping the RS, CELLSRV, and MS services...
dm01cel03: The SHUTDOWN of services was successful.
dm01cel04:
dm01cel04: Stopping the RS, CELLSRV, and MS services...
dm01cel04: The SHUTDOWN of services was successful.
dm01cel05:
dm01cel05: Stopping the RS, CELLSRV, and MS services...
dm01cel05: The SHUTDOWN of services was successful.
dm01cel06:
dm01cel06: Stopping the RS, CELLSRV, and MS services...
dm01cel06: The SHUTDOWN of services was successful.
dm01cel07:
dm01cel07: Stopping the RS, CELLSRV, and MS services...
dm01cel07: The SHUTDOWN of services was successful.
dm01cel08:
dm01cel08: Stopping the RS, CELLSRV, and MS services...
dm01cel08: The SHUTDOWN of services was successful.
dm01cel09:
dm01cel09: Stopping the RS, CELLSRV, and MS services...
dm01cel09: The SHUTDOWN of services was successful.
dm01cel10:
dm01cel10: Stopping the RS, CELLSRV, and MS services...
dm01cel10: The SHUTDOWN of services was successful.
dm01cel11:
dm01cel11: Stopping the RS, CELLSRV, and MS services...
dm01cel11: The SHUTDOWN of services was successful.
dm01cel12:
dm01cel12: Stopping the RS, CELLSRV, and MS services...
dm01cel12: The SHUTDOWN of services was successful.
dm01cel13:
dm01cel13: Stopping the RS, CELLSRV, and MS services...
dm01cel13: The SHUTDOWN of services was successful.
dm01cel14:
dm01cel14: Stopping the RS, CELLSRV, and MS services...
dm01cel14: The SHUTDOWN of services was successful.
[root@dm01db01 ~]#

• Power off all storage cells:

[root@dm01db01 ~]# dcli -g cell_group -l root poweroff
or
[root@dm01db01 ~]# dcli -g cell_group -l root 'shutdown -h now'

• Copy the /root/dbs_group to /root/dbs_group1:

[root@dm01db01 ~]# cp /root/dbs_group /root/dbs_group1

• Remove the "first compute node name" from the /root/dbs_group1 file (using any editor, e.g. vi).

[root@dm01db01 ~]# vi /root/dbs_group1

[root@dm01db01 ~]# cat /root/dbs_group1
dm01db02
dm01db03
dm01db04
dm01db05
dm01db06
dm01db07
dm01db08

• Power off all the compute nodes (except compute node # 1):

[root@dm01db01 ~]# dcli -g dbs_group1 -l root poweroff
or
[root@dm01db01 ~]# dcli -g dbs_group1 -l root 'shutdown -h now'

• Power off the compute node #1:

[root@dm01db01 ~]# poweroff

Broadcast message from root@dm01db01.netsoftmate.com
        (/dev/pts/0) at 7:36 ...

The system is going down for power off NOW!

• Exadata components are shutdown for maintenance.

Conclusion
In this article we have learnt how to Shutdown Exadata Database machine Rack for maintenance.