Overview
You purchased a new Storage cell for expansion or removed Storage from an existing Cluster and want to add it to the new a Cluster due to space demand.
In this article I will demonstrate how to add a storage cell to an
existing Exadata Database Machine.
Here we are
adding new storage cell with 600 GB disks to existing Exadata database machine using
600 GB disks.
Environment
- Exadata Database Machine X2-2 Full Rack
- Exadata Storage Software version 12.1.2.2.0
- Oracle Grid/Database Version 11.2.0.4
Pre Checks:
- Create a new file named "new_cell_group" containing only the hostnames for the new cells.
[root@dm01db01 ~]# vi
new_cell_group
[root@dm01db01 ~]# cat
new_cell_group
dm01cel05
- It is assumed that the file "cell_group" only contains the original cells, not the new ones. If you have already modified "cell_group" to contain the new cells, just comment it out or remove it from the file.
[root@dm01db01 ~]# cat cell_group
dm01cel01
dm01cel02
dm01cel03
dm01cel04
dm01cel05
dm01cel06
dm01cel07
dm01cel08
dm01cel09
dm01cel10
dm01cel11
dm01cel12
dm01cel13
dm01cel14
[root@dm01db01 ~]# vi cell_group
[root@dm01db01 ~]# cat cell_group
dm01cel01
dm01cel02
dm01cel03
dm01cel04
#dm01cel05
dm01cel06
dm01cel07
dm01cel08
dm01cel09
dm01cel10
dm01cel11
dm01cel12
dm01cel13
dm01cel14
- Here we use the root user to run the cellcli commands; ensure that user equivalency setup for root user between Storage cells and Compute nodes.
[root@dm01db01
~]# dcli -g dbs_group -l root 'uptime'
dm01db01:
09:55:05 up 425 days, 23:16, 1
user, load average: 3.26, 3.37, 3.39
dm01db02:
09:55:05 up 117 days, 18:20, 0
users, load average: 1.19, 1.42, 1.56
dm01db03:
09:55:05 up 85 days, 10:07, 0
users, load average: 6.25, 6.20, 6.22
dm01db04:
09:55:05 up 519 days, 11:33, 0
users, load average: 1.53, 1.48, 1.47
dm01db05:
09:55:05 up 519 days, 12:45, 0
users, load average: 1.36, 1.35, 1.47
dm01db06:
09:55:05 up 515 days, 21:40, 0
users, load average: 1.47, 1.36, 1.36
dm01db07:
09:55:05 up 519 days, 12:03, 0
users, load average: 1.44, 1.64, 1.71
dm01db08:
09:55:05 up 519 days, 11:29, 0
users, load average: 1.78, 1.90, 1.78
[root@dm01db01
~]# dcli -g cell_group -l root 'uptime'
dm01cel01:
09:55:15 up 466 days, 20:25, 0
users, load average: 1.44, 1.35, 1.26
dm01cel02:
09:55:15 up 519 days, 14:48, 0
users, load average: 1.49, 1.44, 1.51
dm01cel03:
09:55:15 up 519 days, 14:47, 0
users, load average: 1.01, 0.96, 1.10
dm01cel04:
09:55:15 up 519 days, 14:47, 0
users, load average: 1.40, 1.32, 1.24
dm01cel06:
09:55:15 up 519 days, 14:46, 0
users, load average: 1.10, 1.25, 1.30
dm01cel07:
09:55:15 up 519 days, 14:46, 0
users, load average: 0.99, 1.14, 1.18
dm01cel08:
09:55:15 up 466 days, 19:37, 0
users, load average: 1.52, 1.24, 1.17
dm01cel09:
09:55:15 up 519 days, 14:46, 0
users, load average: 1.00, 1.40, 1.56
dm01cel10:
09:55:15 up 519 days, 14:46, 0
users, load average: 1.09, 1.22, 1.22
dm01cel11:
09:55:15 up 519 days, 14:46, 0
users, load average: 1.24, 1.27, 1.22
dm01cel12:
09:55:15 up 519 days, 14:46, 0
users, load average: 1.11, 1.14, 1.14
dm01cel13:
09:55:15 up 519 days, 14:46, 0
users, load average: 1.13, 1.27, 1.16
dm01cel14:
09:55:15 up 519 days, 14:46, 0
users, load average: 1.20, 1.12, 1.15
Ensure Cell Disks are Ready
- First verify if any cell disks already exist:
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e list celldisk attributes
name,status,freespace where disktype=harddisk"
dm01cel05: CD_00_dm01cel05 normal
0
dm01cel05: CD_01_dm01cel05 normal
0
dm01cel05: CD_02_dm01cel05 normal
0
dm01cel05: CD_03_dm01cel05 normal
0
dm01cel05: CD_04_dm01cel05 normal
0
dm01cel05: CD_05_dm01cel05 normal
0
dm01cel05: CD_06_dm01cel05 normal
0
dm01cel05: CD_07_dm01cel05 normal
0
dm01cel05: CD_08_dm01cel05 normal
0
dm01cel05: CD_09_dm01cel05 normal
0
dm01cel05: CD_10_dm01cel05 normal
0
dm01cel05: CD_11_dm01cel05 normal
0
In my case the cell disks exists.
- If no cell disks are created. There will be no output.
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e list celldisk attributes
name,status,freespace where disktype=harddisk"
[root@dm01db01 ~]#
- If this command returns a list of cell disks, then you should verify if any grid disks exist:
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e list griddisk where
disktype=harddisk"
dm01cel05: DATA_CD_00_dm01cel05 active
dm01cel05: DATA_CD_01_dm01cel05 active
dm01cel05: DATA_CD_02_dm01cel05 active
dm01cel05: DATA_CD_03_dm01cel05 active
dm01cel05: DATA_CD_04_dm01cel05 active
dm01cel05: DATA_CD_05_dm01cel05 active
dm01cel05: DATA_CD_06_dm01cel05 active
dm01cel05: DATA_CD_07_dm01cel05 active
dm01cel05: DATA_CD_08_dm01cel05 active
dm01cel05: DATA_CD_09_dm01cel05 active
dm01cel05: DATA_CD_10_dm01cel05 active
dm01cel05: DATA_CD_11_dm01cel05 active
dm01cel05: RECO_CD_00_dm01cel05 active
dm01cel05: RECO_CD_01_dm01cel05 active
dm01cel05: RECO_CD_02_dm01cel05 active
dm01cel05: RECO_CD_03_dm01cel05 active
dm01cel05: RECO_CD_04_dm01cel05 active
dm01cel05: RECO_CD_05_dm01cel05 active
dm01cel05: RECO_CD_06_dm01cel05 active
dm01cel05: RECO_CD_07_dm01cel05 active
dm01cel05: RECO_CD_08_dm01cel05 active
dm01cel05: RECO_CD_09_dm01cel05 active
dm01cel05: RECO_CD_10_dm01cel05 active
dm01cel05: RECO_CD_11_dm01cel05 active
dm01cel05: DBFS_DG_CD_02_dm01cel05 active
dm01cel05: DBFS_DG_CD_03_dm01cel05 active
dm01cel05: DBFS_DG_CD_04_dm01cel05 active
dm01cel05: DBFS_DG_CD_05_dm01cel05 active
dm01cel05: DBFS_DG_CD_06_dm01cel05 active
dm01cel05: DBFS_DG_CD_07_dm01cel05 active
dm01cel05: DBFS_DG_CD_08_dm01cel05 active
dm01cel05: DBFS_DG_CD_09_dm01cel05 active
dm01cel05: DBFS_DG_CD_10_dm01cel05 active
dm01cel05: DBFS_DG_CD_11_dm01cel05 active
In my case grid disks exists.
- If no grid disks are created. There will be no output.
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e list griddisk where
disktype=harddisk"
[root@dm01db01 ~]#
- If any grid disks are present, then you must ensure they are not in use by looking in V$ASM_DISK and verifying the mount status is "closed" and the header status is "former" or "candidate":
dm01db01-orcldb1 {/home/oracle}:.
oraenv
ORACLE_SID = [orcldb1] ? +ASM1
The Oracle base remains unchanged
with value /u01/app/oracle
dm01db01-+ASM1 {/home/oracle}:sqlplus
/ as sysasm
SQL*Plus: Release 11.2.0.4.0
Production on Mon Oct 17 10:01:15 2016
Copyright (c) 1982, 2013,
Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise
Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application
Clusters and Automatic Storage Management options
SQL> select name, path, state,
mode_status,header_status, mount_status from v$asm_disk where header_status
<> 'MEMBER' order by path;
no rows selected
- If it is OK to drop the grid disks, you can use the following command to drop set of grid disks (by prefix like, RECO):
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e drop griddisk all harddisk
prefix=RECO"
dm01cel05: GridDisk RECO_CD_00_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_01_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_02_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_03_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_04_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_05_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_06_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_07_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_08_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_09_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_10_dm01cel05
successfully dropped
dm01cel05: GridDisk RECO_CD_11_dm01cel05
successfully dropped
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e drop griddisk all harddisk
prefix=DATA"
dm01cel05: GridDisk DATA_CD_00_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_01_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_02_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_03_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_04_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_05_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_06_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_07_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_08_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_09_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_10_dm01cel05
successfully dropped
dm01cel05: GridDisk DATA_CD_11_dm01cel05
successfully dropped
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e drop griddisk all harddisk prefix=DBFS_DG"
dm01cel05: GridDisk DBFS_DG_CD_02_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_03_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_04_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_05_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_06_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_07_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_08_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_09_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_10_dm01cel05
successfully dropped
dm01cel05: GridDisk DBFS_DG_CD_11_dm01cel05
successfully dropped
- Verify Grid disks are dropped
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e list griddisk where
disktype=harddisk"
[root@dm01db01 ~]#
- Drop cell disks
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e drop celldisk all harddisk force"
dm01cel05: CellDisk CD_00_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_01_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_02_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_03_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_04_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_05_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_06_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_07_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_08_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_09_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_10_dm01cel05
successfully dropped
dm01cel05: CellDisk CD_11_dm01cel05
successfully dropped
- If no cell disks exist, create cell disks on all hard disks (output will be similar to the following):
[root@dm01db01 ~]# dcli -g
new_cell_group -l root "cellcli -e create celldisk all harddisk"
dm01cel05: CellDisk CD_00_dm01cel05
successfully created
dm01cel05: CellDisk CD_01_dm01cel05
successfully created
dm01cel05: CellDisk CD_02_dm01cel05
successfully created
dm01cel05: CellDisk CD_03_dm01cel05
successfully created
dm01cel05: CellDisk CD_04_dm01cel05
successfully created
dm01cel05: CellDisk CD_05_dm01cel05
successfully created
dm01cel05: CellDisk CD_06_dm01cel05
successfully created
dm01cel05: CellDisk CD_07_dm01cel05
successfully created
dm01cel05: CellDisk CD_08_dm01cel05
successfully created
dm01cel05: CellDisk CD_09_dm01cel05
successfully created
dm01cel05: CellDisk CD_10_dm01cel05
successfully created
dm01cel05: CellDisk CD_11_dm01cel05
successfully created
- Verify that celldisks were created (output should be similar to the following)
[root@dm01db01 ~]# dcli -g new_cell_group -l root "cellcli -e list celldisk attributes name,status,freespace where disktype=harddisk"
dm01cel05: CD_00_dm01cel05 normal 528.6875G
dm01cel05: CD_01_dm01cel05 normal 528.6875G
dm01cel05: CD_02_dm01cel05 normal 557.8125G
dm01cel05: CD_03_dm01cel05 normal 557.8125G
dm01cel05: CD_04_dm01cel05 normal 557.8125G
dm01cel05: CD_05_dm01cel05 normal 557.8125G
dm01cel05: CD_06_dm01cel05 normal 557.8125G
dm01cel05: CD_07_dm01cel05 normal 557.8125G
dm01cel05: CD_08_dm01cel05 normal 557.8125G
dm01cel05: CD_09_dm01cel05 normal 557.8125G
dm01cel05: CD_10_dm01cel05 normal 557.8125G
dm01cel05: CD_11_dm01cel05 normal 557.8125G
Create DATA and RECO grid disks at the same offsets found on the existing disks (outer tracks)
- Verify the current sizes and offsets for existing grid disks (on the existing cells):
[root@dm01db01
~]# dcli -g cell_group -l root "cellcli -e list griddisk attributes
name,size | grep 'CD_03' | grep -i DBFS_DG"
dm01cel01:
DBFS_DG_CD_03_dm01cel01 29.125G
dm01cel02:
DBFS_DG_CD_03_dm01cel02 29.125G
dm01cel03:
DBFS_DG_CD_03_dm01cel03 29.125G
dm01cel04:
DBFS_DG_CD_03_dm01cel04 29.125G
dm01cel06:
DBFS_DG_CD_03_dm01cel06 29.125G
dm01cel07:
DBFS_DG_CD_03_dm01cel07 29.125G
dm01cel08:
DBFS_DG_CD_03_dm01cel08 29.125G
dm01cel09:
DBFS_DG_CD_03_dm01cel09 29.125G
dm01cel10:
DBFS_DG_CD_03_dm01cel10 29.125G
dm01cel11:
DBFS_DG_CD_03_dm01cel11 29.125G
dm01cel12:
DBFS_DG_CD_03_dm01cel12 29.125G
dm01cel13:
DBFS_DG_CD_03_dm01cel13 29.125G
dm01cel14:
DBFS_DG_CD_03_dm01cel14 29.125G
[root@dm01db01
~]# dcli -g cell_group -l root "cellcli -e list griddisk attributes
name,size,offset | grep 'CD_03' | grep -i data"
dm01cel01:
DATA_CD_03_dm01cel01 260G 32M
dm01cel02:
DATA_CD_03_dm01cel02 260G 32M
dm01cel03:
DATA_CD_03_dm01cel03 260G 32M
dm01cel04:
DATA_CD_03_dm01cel04 260G 32M
dm01cel06:
DATA_CD_03_dm01cel06 260G 32M
dm01cel07:
DATA_CD_03_dm01cel07 260G 32M
dm01cel08:
DATA_CD_03_dm01cel08 260G 32M
dm01cel09:
DATA_CD_03_dm01cel09 260G 32M
dm01cel10:
DATA_CD_03_dm01cel10 260G 32M
dm01cel11:
DATA_CD_03_dm01cel11 260G 32M
dm01cel12:
DATA_CD_03_dm01cel12 260G 32M
dm01cel13:
DATA_CD_03_dm01cel13 260G 32M
dm01cel14:
DATA_CD_03_dm01cel14 260G 32M
[root@dm01db01
~]# dcli -g cell_group -l root "cellcli -e list griddisk attributes
name,size,offset | grep 'CD_03' | grep -i reco"
dm01cel01:
RECO_CD_03_dm01cel01
268.6875G 260.046875G
dm01cel02:
RECO_CD_03_dm01cel02
268.6875G 260.046875G
dm01cel03:
RECO_CD_03_dm01cel03
268.6875G 260.046875G
dm01cel04:
RECO_CD_03_dm01cel04
268.6875G 260.046875G
dm01cel06:
RECO_CD_03_dm01cel06
268.6875G 260.046875G
dm01cel07:
RECO_CD_03_dm01cel07
268.6875G 260.046875G
dm01cel08:
RECO_CD_03_dm01cel08 268.6875G 260.046875G
dm01cel09:
RECO_CD_03_dm01cel09
268.6875G 260.046875G
dm01cel10:
RECO_CD_03_dm01cel10
268.6875G 260.046875G
dm01cel11:
RECO_CD_03_dm01cel11
268.6875G 260.046875G
dm01cel12:
RECO_CD_03_dm01cel12
268.6875G 260.046875G
dm01cel13:
RECO_CD_03_dm01cel13
268.6875G 260.046875G
dm01cel14:
RECO_CD_03_dm01cel14
268.6875G 260.046875G
- Create similar grid disks on the new cells with the same size; be sure to create them in order of offset position, from smallest to largest offset.
[root@dm01db01 ~]# dcli -g new_cell_group -l root "cellcli -e CREATE GRIDDISK ALL HARDDISK PREFIX='DATA', size=260G"
dm01cel05: GridDisk DATA_CD_00_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_01_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_02_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_03_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_04_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_05_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_06_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_07_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_08_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_09_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_10_dm01cel05 successfully created
dm01cel05: GridDisk DATA_CD_11_dm01cel05 successfully created
[root@dm01db01 ~]# dcli -g new_cell_group -l root "cellcli -e CREATE GRIDDISK ALL HARDDISK PREFIX='RECO', size=268.6875G"
dm01cel05: GridDisk RECO_CD_00_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_01_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_02_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_03_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_04_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_05_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_06_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_07_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_08_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_09_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_10_dm01cel05 successfully created
dm01cel05: GridDisk RECO_CD_11_dm01cel05 successfully created
- Validate that grid disks were created properly on the new cells (size and offset matches the original cells)
[root@dm01db01 ~]# dcli -g new_cell_group -l root "cellcli -e list griddisk attributes name,size,offset | grep -i data"
dm01cel05: DATA_CD_00_dm01cel05 260G 32M
dm01cel05: DATA_CD_01_dm01cel05 260G 32M
dm01cel05: DATA_CD_02_dm01cel05 260G 32M
dm01cel05: DATA_CD_03_dm01cel05 260G 32M
dm01cel05: DATA_CD_04_dm01cel05 260G 32M
dm01cel05: DATA_CD_05_dm01cel05 260G 32M
dm01cel05: DATA_CD_06_dm01cel05 260G 32M
dm01cel05: DATA_CD_07_dm01cel05 260G 32M
dm01cel05: DATA_CD_08_dm01cel05 260G 32M
dm01cel05: DATA_CD_09_dm01cel05 260G 32M
dm01cel05: DATA_CD_10_dm01cel05 260G 32M
dm01cel05: DATA_CD_11_dm01cel05 260G 32M
[root@dm01db01 ~]# dcli -g new_cell_group -l root "cellcli -e list griddisk attributes name,size,offset | grep -i reco"
dm01cel05: RECO_CD_00_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_01_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_02_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_03_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_04_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_05_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_06_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_07_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_08_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_09_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_10_dm01cel05 268.6875G 260.046875G
dm01cel05: RECO_CD_11_dm01cel05 268.6875G 260.046875G
Create the new DBFS_DG grid disks only on disks 3 through 12 of the new storage servers
- List the size of the DBFS_DG grid disks on the original cells (they should all be the same size) - we'll call this size, "dbfs_dg_size":
[root@dm01db01 ~]# dcli -c dm01cel01 -l root "cellcli -e list griddisk attributes name,size | grep 'DBFS_DG'"
dm01cel01: DBFS_DG_CD_02_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_03_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_04_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_05_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_06_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_07_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_08_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_09_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_10_dm01cel01 29.125G
dm01cel01: DBFS_DG_CD_11_dm01cel01 29.125G
- The DBFS_DG grid disks will be automatically placed just after the RECO grid disks on the innermost tracks.
NOTE: This command will "fail" on the first and second disks of each cell, but that is expected since there is no more free space on those disks due to the system partitions. The "error" message is similar to: "Cell disks were skipped because they had no freespace for grid disks: CD_00_dm01cel05, CD_01_dm01cel05."
[root@dm01db01 ~]# dcli -g new_cell_group -l root "cellcli -e CREATE GRIDDISK ALL HARDDISK PREFIX='DBFS_DG', size=29.125G"
dm01cel05: Cell disks were skipped because they had no freespace for grid disks: CD_00_dm01cel05, CD_01_dm01cel05.
dm01cel05: GridDisk DBFS_DG_CD_02_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_03_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_04_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_05_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_06_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_07_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_08_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_09_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_10_dm01cel05 successfully created
dm01cel05: GridDisk DBFS_DG_CD_11_dm01cel05 successfully created
Validate that Grid Disks on all Cells are the Same Size
- All Cells at once:
[root@dm01db01 ~]# dcli -g cell_group -l root "cellcli -e list griddisk attributes name,size,offset | grep -i data"
dm01cel01: DATA_CD_00_dm01cel01 260G 32M
dm01cel01: DATA_CD_01_dm01cel01 260G 32M
dm01cel01: DATA_CD_02_dm01cel01 260G 32M
dm01cel01: DATA_CD_03_dm01cel01 260G 32M
dm01cel01: DATA_CD_04_dm01cel01 260G 32M
dm01cel01: DATA_CD_05_dm01cel01 260G 32M
dm01cel01: DATA_CD_06_dm01cel01 260G 32M
dm01cel01: DATA_CD_07_dm01cel01 260G 32M
dm01cel01: DATA_CD_08_dm01cel01 260G 32M
dm01cel01: DATA_CD_09_dm01cel01 260G 32M
dm01cel01: DATA_CD_10_dm01cel01 260G 32M
dm01cel01: DATA_CD_11_dm01cel01 260G 32M
…..
dm01cel14: DATA_CD_00_dm01cel14 260G 32M
dm01cel14: DATA_CD_01_dm01cel14 260G 32M
dm01cel14: DATA_CD_02_dm01cel14 260G 32M
dm01cel14: DATA_CD_03_dm01cel14 260G 32M
dm01cel14: DATA_CD_04_dm01cel14 260G 32M
dm01cel14: DATA_CD_05_dm01cel14 260G 32M
dm01cel14: DATA_CD_06_dm01cel14 260G 32M
dm01cel14: DATA_CD_07_dm01cel14 260G 32M
dm01cel14: DATA_CD_08_dm01cel14 260G 32M
dm01cel14: DATA_CD_09_dm01cel14 260G 32M
dm01cel14: DATA_CD_10_dm01cel14 260G 32M
dm01cel14: DATA_CD_11_dm01cel14 260G 32M
[root@dm01db01 ~]# dcli -g cell_group -l root "cellcli -e list griddisk attributes name,size,offset | grep -i reco"
dm01cel01: RECO_CD_00_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_01_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_02_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_03_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_04_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_05_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_06_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_07_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_08_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_09_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_10_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_11_dm01cel01 268.6875G 260.046875G
……
dm01cel13: RECO_CD_11_dm01cel13 268.6875G 260.046875G
dm01cel14: RECO_CD_00_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_01_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_02_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_03_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_04_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_05_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_06_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_07_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_08_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_09_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_10_dm01cel14 268.6875G 260.046875G
dm01cel14: RECO_CD_11_dm01cel14 268.6875G 260.046875G
[root@dm01db01 ~]# dcli -g cell_group -l root "cellcli -e list griddisk attributes name,size,offset | grep -i DBFS_DG"
dm01cel01: DBFS_DG_CD_02_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_03_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_04_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_05_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_06_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_07_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_08_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_09_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_10_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_11_dm01cel01 29.125G 528.734375G
….
dm01cel14: DBFS_DG_CD_02_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_03_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_04_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_05_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_06_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_07_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_08_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_09_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_10_dm01cel14 29.125G 528.734375G
dm01cel14: DBFS_DG_CD_11_dm01cel14 29.125G 528.734375G
- One cell at a time:
[root@dm01db01 ~]# dcli -c dm01cel01 -l root "cellcli -e list griddisk attributes name,size,offset | grep -i data"
dm01cel01: DATA_CD_00_dm01cel01 260G 32M
dm01cel01: DATA_CD_01_dm01cel01 260G 32M
dm01cel01: DATA_CD_02_dm01cel01 260G 32M
dm01cel01: DATA_CD_03_dm01cel01 260G 32M
dm01cel01: DATA_CD_04_dm01cel01 260G 32M
dm01cel01: DATA_CD_05_dm01cel01 260G 32M
dm01cel01: DATA_CD_06_dm01cel01 260G 32M
dm01cel01: DATA_CD_07_dm01cel01 260G 32M
dm01cel01: DATA_CD_08_dm01cel01 260G 32M
dm01cel01: DATA_CD_09_dm01cel01 260G 32M
dm01cel01: DATA_CD_10_dm01cel01 260G 32M
dm01cel01: DATA_CD_11_dm01cel01 260G 32M
[root@dm01db01 ~]# dcli -c dm01cel01 -l root "cellcli -e list griddisk attributes name,size,offset | grep -i reco"
dm01cel01: RECO_CD_00_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_01_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_02_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_03_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_04_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_05_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_06_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_07_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_08_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_09_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_10_dm01cel01 268.6875G 260.046875G
dm01cel01: RECO_CD_11_dm01cel01 268.6875G 260.046875G
[root@dm01db01 ~]# dcli -c dm01cel01 -l root "cellcli -e list griddisk attributes name,size,offset | grep -i DBFS_DG"
dm01cel01: DBFS_DG_CD_02_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_03_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_04_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_05_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_06_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_07_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_08_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_09_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_10_dm01cel01 29.125G 528.734375G
dm01cel01: DBFS_DG_CD_11_dm01cel01 29.125G 528.734375G
Add the DBFS_DG grid disks to the existing DBFS_DG disk group
- Update the cellip.ora file
[root@dm01db01 ~]# vi /etc/oracle/cell/network-config/cellip.ora
[root@dm01db01 network-config]# cat /etc/oracle/cell/network-config/cellip.ora
cell="192.168.2.9"
cell="192.168.2.10"
cell="192.168.2.11"
cell="192.168.2.12"
cell="192.168.2.13"
cell="192.168.2.14"
cell="192.168.2.15"
cell="192.168.2.16"
cell="192.168.2.17"
cell="192.168.2.18"
cell="192.168.2.19"
cell="192.168.2.20"
cell="192.168.2.21"
cell="192.168.2.22"
[root@dm01db01 ~]# cd /etc/oracle/cell/network-config/
[root@dm01db01 network-config]# dcli -g ~/dbs_group -l root -d /etc/oracle/cell/network-config -f cellip.ora
[root@dm01db01 network-config]# dcli -g ~/dbs_group -l root ls -l /etc/oracle/cell/network-config/cellip.ora
dm01db01: -rwxr-xr-x 1 oracle oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db02: -rwx------ 1 oracle oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db03: -rwx------ 1 oracle oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db04: -rwx------ 1 oracle oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db05: -rwx------ 1 oracle oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db06: -rwx------ 1 oracle oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db07: -rwx------ 1 oracle oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db08: -rwx------ 1 oracle oinstall 279 Oct 17 11:15 /etc/oracle/cell/network-config/cellip.ora
dm01db01-orcldb1 {/home/oracle}:. oraenv
ORACLE_SID = [orcldb1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
dm01db01-+ASM1 {/home/oracle}:sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Mon Oct 17 11:18:34 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> alter diskgroup DBFS_DG add disk 'o/*/DBFS_DG*dm01cel05' rebalance power 32 NOWAIT;
Diskgroup altered.
Add the DATA and RECO grid disks to the existing DATA and RECO disk groups
- From one DB node add the new DATA disks to the DATA disk group (replace the pattern below " 'o/*/DATA_DM01*new_cel01', 'o/*/DATA_DM01*new_cel02', 'o/*/DATA_DM01*new_cel03'" with list of actual new cells):
SQL> alter diskgroup DATA add disk 'o/*/DATA*dm01cel05' rebalance power 32 NOWAIT;
Diskgroup altered.
- From another DB node add the new RECO disks to the RECO disk group (replace the pattern below, " 'o/*/RECO_DM01*new_cel01','o/*/RECO_DM01*new_cel02','o/*/RECO_DM01*new_cel03' " with list of actual new cells):
SQL> alter diskgroup RECO add disk 'o/*/RECO*dm01cel05' rebalance power 32 NOWAIT;
Diskgroup altered.
- Monitor the progress of the rebalance activities using this query (you can proceed with the rest of the steps while this is continuing):
SQL> select * from gv$asm_operation;
INST_ID GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
---------- ------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
7 1 REBAL WAIT 32
3 1 REBAL WAIT 32
2 1 REBAL WAIT 32
1 1 REBAL WAIT 32
6 1 REBAL WAIT 32
8 1 REBAL WAIT 32
4 1 REBAL WAIT 32
5 1 REBAL RUN 32 32 41666 45014 17263 0
8 rows selected.
SQL> select * from gv$asm_operation;
no rows selected
Validate that DATA, RECO, and DBFS_DG contain disks from new cells
Before:
dm01db01-orcldb1 {/home/oracle}:. oraenv
ORACLE_SID = [orcldb1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
dm01db01-+ASM1 {/home/oracle}:sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Mon Oct 17 10:43:16 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set lines 200
SQL> set pages 200
SQL> select d.failgroup,dg.name, sum(d.total_mb) cell_total_mb, count(1) cell_disk_count from v$asm_disk d, v$asm_diskgroup dg where d.group_number = dg.group_number group by d.failgroup, dg.name order by d.failgroup, dg.name;
FAILGROUP NAME CELL_TOTAL_MB CELL_DISK_COUNT
------------------------------ ------------------------------ ------------- ---------------
DM01CEL01 DATA 3194880 12
DM01CEL01 RECO 3301632 12
DM01CEL01 DBFS_DG 298240 10
DM01CEL02 DATA 2928640 12
DM01CEL02 RECO 3026496 12
DM01CEL02 DBFS_DG 268416 10
DM01CEL03 DATA 3194880 12
DM01CEL03 RECO 3301632 12
DM01CEL03 DBFS_DG 298240 10
DM01CEL04 DATA 3194880 12
DM01CEL04 RECO 3301632 12
DM01CEL04 DBFS_DG 298240 10
DM01CEL06 DATA 3194880 12
DM01CEL06 RECO 3301632 12
DM01CEL06 DBFS_DG 298240 10
DM01CEL07 DATA 2928640 11
DM01CEL07 RECO 3026496 11
DM01CEL07 DBFS_DG 268416 10
DM01CEL08 DATA 3194880 12
DM01CEL08 RECO 3301632 12
DM01CEL08 DBFS_DG 298240 10
DM01CEL09 DATA 2396160 12
DM01CEL09 RECO 2476224 12
DM01CEL09 DBFS_DG 238592 10
DM01CEL10 DATA 3194880 12
DM01CEL10 RECO 3301632 12
DM01CEL10 DBFS_DG 298240 10
DM01CEL11 DATA 2928640 12
DM01CEL11 RECO 3026496 12
DM01CEL11 DBFS_DG 298240 10
DM01CEL12 DATA 3194880 12
DM01CEL12 RECO 3301632 12
DM01CEL12 DBFS_DG 298240 10
DM01CEL13 DATA 3194880 12
DM01CEL13 RECO 3301632 12
DM01CEL13 DBFS_DG 298240 10
DM01CEL14 DATA 3194880 12
DM01CEL14 RECO 3301632 12
DM01CEL14 DBFS_DG 298240 10
39 rows selected.
After
SQL> set lines 200
SQL> set pages 200
SQL> select d.failgroup,dg.name, sum(d.total_mb) cell_total_mb, count(1) cell_disk_count from v$asm_disk d, v$asm_diskgroup dg where d.group_number = dg.group_number group by d.failgroup, dg.name order by d.failgroup, dg.name;
FAILGROUP NAME CELL_TOTAL_MB CELL_DISK_COUNT
------------------------------ ------------------------------ ------------- ---------------
DM01CEL01 DATA 3194880 12
DM01CEL01 RECO 3301632 12
DM01CEL01 DBFS_DG 298240 10
DM01CEL02 DATA 2928640 12
DM01CEL02 RECO 3026496 12
DM01CEL02 DBFS_DG 268416 10
DM01CEL03 DATA 3194880 12
DM01CEL03 RECO 3301632 12
DM01CEL03 DBFS_DG 298240 10
DM01CEL04 DATA 2928640 12
DM01CEL04 RECO 3026496 12
DM01CEL04 DBFS_DG 268416 10
DM01CEL05 DATA 3194880 12
DM01CEL05 RECO 3301632 12
DM01CEL05 DBFS_DG 298240 10
DM01CEL06 DATA 3194880 12
DM01CEL06 RECO 3301632 12
DM01CEL06 DBFS_DG 298240 10
DM01CEL07 DATA 2928640 12
DM01CEL07 RECO 3026496 12
DM01CEL07 DBFS_DG 268416 10
DM01CEL08 DATA 3194880 12
DM01CEL08 RECO 3301632 12
DM01CEL08 DBFS_DG 298240 10
DM01CEL09 DATA 2396160 12
DM01CEL09 RECO 2476224 12
DM01CEL09 DBFS_DG 238592 10
DM01CEL10 DATA 3194880 12
DM01CEL10 RECO 3301632 12
DM01CEL10 DBFS_DG 298240 10
DM01CEL11 DATA 2928640 12
DM01CEL11 RECO 3026496 12
DM01CEL11 DBFS_DG 298240 10
DM01CEL12 DATA 3194880 12
DM01CEL12 RECO 3301632 12
DM01CEL12 DBFS_DG 298240 10
DM01CEL13 DATA 3194880 12
DM01CEL13 RECO 3301632 12
DM01CEL13 DBFS_DG 298240 10
DM01CEL14 DATA 3194880 12
DM01CEL14 RECO 3301632 12
DM01CEL14 DBFS_DG 298240 10
42 rows selected.
Ensure that there are no offline
disks in any disk group:
SQL> select
name,total_mb,free_mb,offline_disks from v$asm_diskgroup;
NAME TOTAL_MB FREE_MB OFFLINE_DISKS
------------------------------
---------- ---------- -------------
DATA 42864640 41713708 0
RECO 44296896 44293768 0
DBFS_DG 4026240 4023232 0
Conclusion
In this article we have learned
how to add a storage cell to an existing Exadata Database machine of same disk
size.
No comments:
Post a Comment