通过udev创建ASM共享磁盘（RAC）

131次阅读

共计 9862 个字符，预计需要花费 25 分钟才能阅读完成。

OS：RedHat EL6.0

Oracle：Oracle 11gR2

在 Oracle 11gR2，构建 RAC 时可以通过 ASM 创建 asm disk，但是需要安装 asmlib 相关软件；对于 RedHat EL6.0，Oracle 的 asm 不能很好的支持，所以可以通过 udev 来创建 asm disk，就不需要安装 asmlib 相关软件。

一、查看系统环境

内核版本：

[root@rac01 rules.d]# uname -a

Linux rac01 2.6.32-642.6.2.el6.x86_64 #1 SMP Wed Oct 26 06:52:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

系统用户：

[root@rac01 rules.d]# id grid

uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1300(dba),1200(asmadmin),1201(asmdba),1202(asmoper)

二、创建共享磁盘

在 Windows server 2012R2 服务器上启用 iSCSI 服务，并新建 iSCSI 共享磁盘

通过 udev 创建 ASM 共享磁盘（RAC）

选择有足够容量的卷通过 udev 创建 ASM 共享磁盘（RAC）

给虚拟磁盘命名并划分大小，然后配置 iSCSI 目标通过 udev 创建 ASM 共享磁盘（RAC）

命名访问目标，添加访问程序，选择 ”IP地址类型 ”，输入 RAC 节点 IP（将两个节点的IP 都添加上）通过 udev 创建 ASM 共享磁盘（RAC）

继续下一步，不启用验证服务，确认创建，创建完成后会在管理页面显示出已经创建好的共享磁盘通过 udev 创建 ASM 共享磁盘（RAC）

三、配置共享存储

在节点 1 和节点 2 上分别安装 iSCSI Initiator，以节点一为例，主要涉及以下命令

yum -y install iscsi-initiator-utils

chkconfig iscsid on

chkconfig iscsi on

最后发现并且挂载 iscsi target

[root@rac01 rules.d]# iscsiadm -m discovery -t sendtargets -p 192.168.2.70

192.168.2.70:3260,1 iqn.1991-05.com.microsoft:win-ct207djdp08-max-002-target

192.168.2.70:3260,1 iqn.1991-05.com.microsoft:win-ct207djdp08-test1-target

[root@rac01 rules.d]# iscsiadm -m node -T iqn.1991-05.com.microsoft:win-ct207djdp08-max-002-target –login

成功后，用 fdisk -l 命令可以看到多出的共享磁盘，进行分区，分区结果如下图通过 udev 创建 ASM 共享磁盘（RAC）

四、建立 UDEV 规则文件

查看磁盘UUID：

[root@rac01 rules.d]# /sbin/scsi_id -g -u -d /dev/sdb

360003ff44dc75adca44127e408e97955

创建 UDEV 配置文件：

[root@rac01 rules.d]# cd /etc/udev/rules.d/

[root@rac01 rules.d]# vim 99-oracle-asmdisk.rules

KERNEL==”sdb5″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”360003ff44dc75adca44127e408e97955″, NAME=”asm_disk1″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

KERNEL==”sdb6″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”360003ff44dc75adca44127e408e97955″, NAME=”asm_disk2″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

KERNEL==”sdb7″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”360003ff44dc75adca44127e408e97955″, NAME=”asm_disk3″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

KERNEL==”sdb8″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”360003ff44dc75adca44127e408e97955″, NAME=”asm_disk4″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

KERNEL==”sdb9″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”360003ff44dc75adca44127e408e97955″, NAME=”asm_disk5″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

KERNEL==”sdb10″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”360003ff44dc75adca44127e408e97955″, NAME=”asm_disk6″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

五、启动UDEV service

载入配置文件：

[root@rac01 rules.d]# udevadm control –reload-rules

启动UDEV service:

[root@rac01 rules.d]# start_udev

查看asm disks:

[root@rac01 rules.d]# ls -l /dev/asm*

brw-rw—- 1 grid asmadmin 8, 21 Nov 25 11:22 /dev/asm_disk1

brw-rw—- 1 grid asmadmin 8, 22 Nov 25 11:22 /dev/asm_disk2

brw-rw—- 1 grid asmadmin 8, 23 Nov 25 11:22 /dev/asm_disk3

brw-rw—- 1 grid asmadmin 8, 24 Nov 25 11:22 /dev/asm_disk4

brw-rw—- 1 grid asmadmin 8, 25 Nov 25 11:22 /dev/asm_disk5

brw-rw—- 1 grid asmadmin 8, 26 Nov 25 11:22 /dev/asm_disk6

六、添加额外 ASM 磁盘

安装完 RAC 环境后，突然觉得之前配置的 ASM 磁盘不够用，于是打算再添加一块大容量的 ASM 磁盘；前面操作都无差异，这里补充后续操作。

查看asm disks:

[root@rac01 rules.d]# ls -l /dev/asm*

brw-rw—- 1 grid asmadmin 8, 21 Nov 25 11:22 /dev/asm_disk1

brw-rw—- 1 grid asmadmin 8, 22 Nov 25 11:22 /dev/asm_disk2

brw-rw—- 1 grid asmadmin 8, 23 Nov 25 11:22 /dev/asm_disk3

brw-rw—- 1 grid asmadmin 8, 24 Nov 25 11:22 /dev/asm_disk4

brw-rw—- 1 grid asmadmin 8, 25 Nov 25 11:22 /dev/asm_disk5

brw-rw—- 1 grid asmadmin 8, 26 Nov 25 11:22 /dev/asm_disk6

brw-rw—- 1 grid asmadmin 8, 33 Nov 25 09:42 /dev/asm_disk7

在节点 1 中，使用 grid 用户运行asmca: 通过 udev 创建 ASM 共享磁盘（RAC）

成功弹出 ASM 设置界面，选择Create

输入磁盘组名，选择External（外部冗余）（因为已经存在常规冗余磁盘组，为了省事儿这里选择外部冗余），选择磁盘：通过 udev 创建 ASM 共享磁盘（RAC）

点击OK

通过 udev 创建 ASM 共享磁盘（RAC）

点击 OK 回到初始界面，点击右下角 Exit，然后选择Yes 保留操作并退出

使用 oracle 账户登录数据库并查询 ASM 磁盘组信息：

[root@rac01 ~]# su – oracle

[oracle@rac01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:09:54 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select name,total_mb from v$asm_diskgroup;

NAME TOTAL_MB

—————————— ———-

DATA_DB 16378

DATA 6144

CY 102398

七、额外问题

完成上面的操作后，发现节点 2 出现了一些问题

[oracle@rac02 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:13:45 2016

Connected to an idle instance.

SQL> select name,total_mb from v$asm_diskgroup;

select name,total_mb from v$asm_diskgroup

*

ERROR at line 1:

ORA-01034: ORACLE not available

Process ID: 0

Session ID: 0 Serial number: 0

实例似乎被关闭了，重启实例后，发现新加的 ASM 磁盘组挂载异常

SQL> startup

ORACLE instance started.

Total System Global Area 1603411968 bytes

Fixed Size 2253664 bytes

Variable Size 1056967840 bytes

Database Buffers 536870912 bytes

Redo Buffers 7319552 bytes

Database mounted.

Database opened.

SQL> select name,total_mb from v$asm_diskgroup;

NAME TOTAL_MB

—————————— ———-

DATA_DB 16378

DATA 6144

CY 0

[grid@rac02 admin]$ ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 3

Total space (kbytes) : 262120

Used space (kbytes) : 3072

Available space (kbytes) : 259048

ID : 277519239

Device/File Name : +DATA

Device/File integrity check succeeded

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

手动挂载：

[grid@rac02 admin]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:30:30 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production

With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup CY mount;

Diskgroup altered.

SQL> select GROUP_NUMBER,NAME,TYPE,ALLOCATION_UNIT_SIZE,STATE from v$asm_diskgroup;

GROUP_NUMBER NAME TYPE ALLOCATION_UNIT_SIZE STATE

———— —————————— —— ——————–

1 DATA_DB NORMAL 1048576 MOUNTED

2 DATA NORMAL 1048576 MOUNTED

3 CY EXTERN 1048576 MOUNTED

再次查询 ASM 磁盘组信息：

[oracle@rac02 admin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:40:10 2016

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select name,total_mb from v$asm_diskgroup;

NAME TOTAL_MB

—————————— ———-

DATA_DB 16378

DATA 6144

CY 102398

完成后发现 CRS 服务异常：

[grid@rac02 admin]$ crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4534: Cannot communicate with Event Manager

排查后发现是硬盘名称发生了变化，修改完相关配置文件（vim /etc/udev/rules.d/99-oracle-asmdisk.rules），CRS服务正常：

[grid@rac02 ~]$ crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

然后发现因为之前的异常操作，导致监听异常：

[grid@rac02 admin]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 – Production on 25-NOV-2016 15:19:25

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))

STATUS of the LISTENER

————————

Alias LISTENER

Version TNSLSNR for Linux: Version 11.2.0.4.0 – Production

Start Date 25-NOV-2016 15:11:22

Uptime 0 days 0 hr. 8 min. 2 sec

Trace Level off

Security ON: Local OS Authentication

SNMP OFF

Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora

Listener Log File /u01/app/11.2.0/grid/log/diag/tnslsnr/rac02/listener/alert/log.xml

Listening Endpoints Summary…

(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.209)(PORT=1521)))

Services Summary…

Service “+ASM” has 1 instance(s).

Instance “+ASM2”, status READY, has 1 handler(s) for this service…

Service “rac” has 1 instance(s).

Instance “rac2”, status READY, has 1 handler(s) for this service…

Service “racXDB” has 1 instance(s).

Instance “rac2”, status READY, has 1 handler(s) for this service…

The command completed successfully

以节点 1 为例

修改两个节点的监听文件：

[root@rac01 ~]# su – grid

[grid@rac01 ~]$ cd /u01/app/11.2.0/grid/network/admin

[grid@rac01 admin]$ ls

endpoints_listener.ora listener.ora shrept.lst

endpoints_listener.ora.bak.rac01 listener.ora.bak.rac01 sqlnet.ora

listener16112111AM0504.bak samples

[grid@rac01 admin]$ vim endpoints_listener.ora

#将 HOST 参数改为本机IP

LISTENER_RAC01=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.2.206)(PORT=1521)(IP=FIRST))))

修改 TNS 配置文件：

[oracle@rac01 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/network/admin

[oracle@rac01 admin]$ ls

samples shrept.lst tnsnames.ora

[oracle@rac01 admin]$ vim tnsnames.ora

RAC =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = max-cluster)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = rac)

)