Multipath on Red Hat Linux

Discover ISCSI storage :

[root@test01 ~]# fdisk -l
Disk /dev/sda: 146.1 GB, 146163105792 bytes
255 heads, 63 sectors/track, 17769 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14       17769   142625070   8e  Linux LVM

<<No SAN storage is showing up., above is local storage>>

-Check to see if ISCSI service is running

     # service iscsi status
          iscsid (pid  6094) is running...

-By default ISCSI initiator is not enabled during boot. Enter the following command to enable at boot.

   #chkconfig --add iscsi
   #chkconfig iscsi on
   #chkconfig --list iscsi

-Discover ISCSI Storage target.

     iscsiadm -m discovery -t st -p <<ISCSI storage ipaddress>>

Example :

        #iscsiadm -m discovery -t st -p 10.0.5.50

-Connect to iscsi targets

      #iscsiadm -m node -l

You should see ISCSI volumes in the output.

-Check to see if both adaptors connected to array

    #iscsiadm -m session
      tcp: [3] 10.0.5.50:3260,1 iqn.2001-05.com....
      tcp: [4] 10.0.5.50:3260,1 iqn.2001-05.com...

-Restart the iscsi service.

    #service iscsi restart

    #fdisk -l
   Disk /dev/sda: 146.1 GB, 146163105792 bytes
   255 heads, 63 sectors/track, 17769 cylinders
  Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
  /dev/sda1   *           1          13      104391   83  Linux
  /dev/sda2              14       17769   142625070   8e  Linux LVM

   Disk /dev/sdb: 214.7 GB, 214758850560 bytes                 <--- ISCSI Storage   255 heads, 63 sectors/track, 26109 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes  
   Device Boot      Start         End      Blocks   Id  System

Setup Mulipathing:

Setup multipath of (/dev/sdb) through eth6 and eth7 interfaces.

- Create an interface file for each network interface.

#iscsiadm -m iface -l eth6 -new
New interface eth6 added

#iscsiadm -m iface -l eth7 -new
New interface eth7 added

-Update interface name
#iscsiadm -m iface -I eth6 --op=update -n iface.net_ifacename -v eth6
#iscsiadm -m iface -I eth7 --op=update -n iface.net_ifacename -v eth7

It will create 2 files( eth6,eth7) under /var/lib/iscsi/ifaces/

-Discover your storage volumes again

#iscsiadm -m discovery -t st -p 10.0.5.50

-Check your disks and you should see each volume disk twice.

#fdisk -l

Disk /dev/sda: 146.1 GB, 146163105792 bytes
255 heads, 63 sectors/track, 17769 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14       17769   142625070   8e  Linux LVM

Disk /dev/sdb: 214.7 GB, 214758850560 bytes
255 heads, 63 sectors/track, 26109 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System      <--- Same disk twice means multipath enabled.

Disk /dev/sdc: 214.7 GB, 214758850560 bytes
255 heads, 63 sectors/track, 26109 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
   Device Boot      Start         End      Blocks   Id  System


After reboot there is no guarentee that you will get above disk as /dev/sda. To address this you need use persistent device naming, before that to verify if multipath is configured correctly try following commands.

#multipath -v2
#multipath -ll
#

The above not showing any output. It means multipathing is not enabled. modify /etc/multipath.conf to enable mulipathing.

modify the following line in "mulipath.conf" file to enable mulipathing on /dev/sd? devices.

from

       blacklist {
        devnode "*"
       }

To

blacklist {
        devnode "^sd$"
}
now run the following command again to verify multipathing.

#multipath -v2
#multipath -ll
mpath6 (3560xty456000d6b31415c59a01b081) dm-6 EQLOGIC,100E-00
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 12:0:0:0 sdb 8:16  [active][ready]
 \_ 13:0:0:0 sdc 8:32  [active][ready]

Multipath is enabled. Now set the persistent naming so whenever system boots we get the same device mapping. The UUID from above output never changes and we will use UUID for persistent name.

Open multipath.conf file and go to the multipath section and make the following changes.

multipaths {
       multipath {
               wwid                    3560xty456000d6b31415c59a01b081    <-- UUID from above
               alias                      backupdisk                                               <-- friendly name for the device
               path_grouping_policy    multibus
               #path_checker            readsector0
               path_selector           "round-robin 0"
               failback                manual
               rr_weight               priorities
               no_path_retry           5
               rr_min_io               15
       }
    

#

#multipath -v2
#multipath -ll
backupdisk (3560xty456000d6b31415c59a01b081) dm-2 EQLOGIC,100E-00
[size=200G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 11:0:0:0 sdb 8:16  [active][ready]
 \_ 12:0:0:0 sdc 8:32  [active][ready]


The above outout shows the new label to backupdisk

#ls -l /dev/mapper

brw-rw---- 1 root disk 253,  2 Mar 22 11:55 backupdisk
crw------- 1 root root  10, 60 Mar 22 10:34 control
backupdisk is now a persistent name for /dev/sda or /dev/sdb devices.

-Create and mount filesystem on /dev/mapper/backupdisk

# mkfs.ext3 /dev/mapper/backupdisk
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
26230784 inodes, 52431360 blocks
2621568 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
1601 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872
Writing inode tables: done
# mkdir /backup

edit the /etc/fstab and make add entry for /dev/mapper/backupdisk to mount on /backup after boot.

/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0
/dev/mapper/backupdisk   /backup                 ext3    defaults        0 0
~
#mount /dev/mapper/backupdisk
#df -k
# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      69893520   2446488  63839276   4% /
/dev/sda1               101086     12952     82915  14% /boot
tmpfs                 49477868         0  49477868   0% /dev/shm
/dev/mapper/backupdisk
                     206432944    191892 195754780   1% /backup

- make sure to add these setting into /etc/rc.d/rc.local file to preserve these setting after restart.

The file should look like this :


touch /var/lock/subsys/local
/sbin/modprobe bonding
/sbin/service network restart
multipath -v2
multipath -ll
chown oracle:oinstall /dev/mapper/data*
chown oracle:oinstall /dev/mapper/back*
mount /backup