- I start with rebooting the server, to have a clean/known state to start with. - iSCSI LUNs are set up automatically during system startup. - multipathd is also started automatically during system startup. - For this test I'm using open-iscsi software initiator included with RHEL5. /dev/sdc is from a path accessed through open-iscsi "iface0", and /dev/sdd is from a path accessed through open-iscsi "iface1". - (Although the system/root disk is on dual port Qlogic qla4xxx iSCSI HBA and is also using multipath (RHEL5 installed with "linux mpath")) Originally I tested with RHEL 5.2 (with some new beta packages to make this possible): # uname -r 2.6.18-93.el5.bz444964.2 # rpm -qa|grep device-mapper-multipath device-mapper-multipath-0.4.7-19.el5 Later I verified this to work with RHEL 5.3 GA (which includes the changes to make this possible): # uname -r 2.6.18-128.el5PAE # rpm -qa|grep device-mapper-multipath device-mapper-multipath-0.4.7-23.el5 # cat /proc/partitions major minor #blocks name 8 0 10490880 sda 8 1 257008 sda1 8 2 10233405 sda2 8 16 10490880 sdb 8 17 257008 sdb1 8 18 10233405 sdb2 253 0 10490880 dm-0 253 1 257008 dm-1 253 2 10233405 dm-2 253 3 8126464 dm-3 253 4 2097152 dm-4 8 48 2104320 sdd 8 32 2104320 sdc 253 5 2104320 dm-5 # blockdev --getsize /dev/sdc 4208640 # blockdev --getsize /dev/sdd 4208640 # multipath -ll mpath0 (30690a018c032c49fee6364140000d06d) dm-0 EQLOGIC,100E-00 [size=10G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 1:0:0:0 sdb 8:16 [active][ready] mpath-resize-test (36090a018c032e4801e8e341700008039) dm-5 EQLOGIC,100E-00 [size=2.0G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 2:0:0:0 sdd 8:48 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 3:0:0:0 sdc 8:32 [active][ready] # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-lvroot 7.6G 2.0G 5.2G 28% / /dev/mapper/mpath0p1 244M 15M 216M 7% /boot tmpfs 1.7G 0 1.7G 0% /dev/shm # cat /etc/multipath.conf defaults { user_friendly_names yes } blacklist { devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^(hd|xvd)[a-z]*" wwid "*" } # Make sure our multipath devices are enabled. blacklist_exceptions { wwid "30690a018c032c49fee6364140000d06d" wwid "36090a018c032e4801e8e341700008039" } multipaths { multipath { wwid "36090a018c032e4801e8e341700008039" alias mpath-resize-test } } # ps aux|grep -i multi root 2822 0.0 0.0 4408 3404 ? SLl 15:18 0:00 /sbin/multipathd -v3 root 3852 0.0 0.0 3912 728 pts/2 S+ 15:21 0:00 grep -i multi # dmsetup info mpath-resize-test Name: mpath-resize-test State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 0 Event number: 1 Major, minor: 253, 5 Number of targets: 1 UUID: mpath-36090a018c032e4801e8e341700008039 # multipathd -k"show paths" hcil dev dev_t pri dm_st chk_st next_check 0:0:0:0 sda 8:0 1 [active][ready] XXX....... 6/20 1:0:0:0 sdb 8:16 1 [active][ready] XXX....... 6/20 3:0:0:0 sdc 8:32 1 [active][ready] XXX....... 6/20 2:0:0:0 sdd 8:48 1 [active][ready] XXX....... 6/20 # multipathd -k"show devices" available block devices: dm-0 (blacklisted) dm-1 (blacklisted) dm-2 (blacklisted) dm-3 (blacklisted) dm-4 (blacklisted) dm-5 (blacklisted) md0 (blacklisted) ram0 (blacklisted) ram10 (blacklisted) ram11 (blacklisted) ram12 (blacklisted) ram13 (blacklisted) ram14 (blacklisted) ram15 (blacklisted) ram1 (blacklisted) ram2 (blacklisted) ram3 (blacklisted) ram4 (blacklisted) ram5 (blacklisted) ram6 (blacklisted) ram7 (blacklisted) ram8 (blacklisted) ram9 (blacklisted) sda sdb sdc sdd # multipathd -k"show maps" name sysfs uuid mpath0 dm-0 30690a018c032c49fee6364140000d06d mpath-resize-test dm-5 36090a018c032e4801e8e341700008039 # mkfs.ext3 /dev/mapper/mpath-resize-test # mkdir /mnt/testvol # mount /dev/mapper/mpath-resize-test /mnt/testvol # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-lvroot 7.6G 2.0G 5.2G 28% / /dev/mapper/mpath0p1 244M 15M 216M 7% /boot tmpfs 1.7G 0 1.7G 0% /dev/shm /dev/mapper/mpath-resize-test 2.0G 68M 1.9G 4% /mnt/testvol ------------------------------------------------------------------------------ And at this point I resize the LUN on (Equallogic iSCSI) SAN from 2 GB to 4 GB. ------------------------------------------------------------------------------ # Rescan iSCSI LUNs to pick up the new size # iscsiadm -m node -R -I iface0 # iscsiadm -m node -R -I iface1 # dmesg|tail -10 SCSI device sdd: 8417280 512-byte hdwr sectors (4310 MB) sdd: Write Protect is off sdd: Mode Sense: 81 00 00 00 SCSI device sdd: drive cache: write through sdd: detected capacity change from 2154823680 to 4309647360 SCSI device sdc: 8417280 512-byte hdwr sectors (4310 MB) sdc: Write Protect is off sdc: Mode Sense: 81 00 00 00 SCSI device sdc: drive cache: write through sdc: detected capacity change from 2154823680 to 4309647360 # blockdev --getsize /dev/sdc 8417280 # blockdev --getsize /dev/sdd 8417280 # cat /proc/partitions major minor #blocks name 8 0 10490880 sda 8 1 257008 sda1 8 2 10233405 sda2 8 16 10490880 sdb 8 17 257008 sdb1 8 18 10233405 sdb2 253 0 10490880 dm-0 253 1 257008 dm-1 253 2 10233405 dm-2 253 3 8126464 dm-3 253 4 2097152 dm-4 8 48 4208640 sdd 8 32 4208640 sdc 253 5 2104320 dm-5 # So now sdc and sdd are clearly resized from 2 GB to 4 GB # Time to try mpath device resizing.. # multipath -ll mpath0 (30690a018c032c49fee6364140000d06d) dm-0 EQLOGIC,100E-00 [size=10G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 1:0:0:0 sdb 8:16 [active][ready] mpath-resize-test (36090a018c032e4801e8e341700008039) dm-5 EQLOGIC,100E-00 [size=2.0G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 2:0:0:0 sdd 8:48 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 3:0:0:0 sdc 8:32 [active][ready] # multipathd -k multipathd> del path sdc ok multipathd> add path sdc ok multipathd> del path sdd ok multipathd> add path sdd ok multipathd> resize map mpath-resize-test ok UPDATE 2009-03-23: With RHEL 5.3 GA it's only required to run the final "resize map" command with "multipathd -k". # multipath -ll mpath0 (30690a018c032c49fee6364140000d06d) dm-0 EQLOGIC,100E-00 [size=10G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 1:0:0:0 sdb 8:16 [active][ready] mpath-resize-test (36090a018c032e4801e8e341700008039) dm-5 EQLOGIC,100E-00 [size=4.0G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][enabled] \_ 3:0:0:0 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 2:0:0:0 sdd 8:48 [active][ready] # resize2fs /dev/mapper/mpath-resize-test resize2fs 1.39 (29-May-2006) Filesystem at /dev/mapper/mpath-resize-test is mounted on /mnt/testvol; on-line resizing required Performing an on-line resize of /dev/mapper/mpath-resize-test to 1052160 (4k) blocks. The filesystem on /dev/mapper/mpath-resize-test is now 1052160 blocks long. # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-lvroot 7.6G 2.0G 5.2G 28% / /dev/mapper/mpath0p1 244M 15M 216M 7% /boot tmpfs 1.7G 0 1.7G 0% /dev/shm /dev/mapper/mpath-resize-test 4.0G 69M 3.7G 2% /mnt/testvol ---------------------------------------------------------------- OK, It seems to work! Let's try LVM PV/VG on top of mpath device ---------------------------------------------------------------- # umount /mnt/testvol # pvcreate /dev/mapper/mpath-resize-test Physical volume "/dev/mapper/mpath-resize-test" successfully created # vgcreate vgtest /dev/mapper/mpath-resize-test Volume group "vgtest" successfully created # lvcreate -ntestvol -L4G /dev/vgtest Logical volume "testvol" created # mkfs.ext3 /dev/vgtest/testvol # mount /dev/vgtest/testvol /mnt/testvol # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-lvroot 7.6G 2.0G 5.2G 28% / /dev/mapper/mpath0p1 244M 15M 216M 7% /boot tmpfs 1.7G 0 1.7G 0% /dev/shm /dev/mapper/vgtest-testvol 4.0G 137M 3.7G 4% /mnt/testvol ------------------------------------------------------------------------------ And at this point I resize the LUN on (Equallogic iSCSI) SAN from 4 GB to 6 GB. ------------------------------------------------------------------------------ # Rescan iSCSI LUNs to pick up the new size # iscsiadm -m node -R -I iface0 # iscsiadm -m node -R -I iface1 # dmesg|tail -10 SCSI device sdd: 12595200 512-byte hdwr sectors (6449 MB) sdd: Write Protect is off sdd: Mode Sense: 81 00 00 00 SCSI device sdd: drive cache: write through sdd: detected capacity change from 4309647360 to 6448742400 SCSI device sdc: 12595200 512-byte hdwr sectors (6449 MB) sdc: Write Protect is off sdc: Mode Sense: 81 00 00 00 SCSI device sdc: drive cache: write through sdc: detected capacity change from 4309647360 to 6448742400 # cat /proc/partitions major minor #blocks name 8 0 10490880 sda 8 1 257008 sda1 8 2 10233405 sda2 8 16 10490880 sdb 8 17 257008 sdb1 8 18 10233405 sdb2 253 0 10490880 dm-0 253 1 257008 dm-1 253 2 10233405 dm-2 253 3 8126464 dm-3 253 4 2097152 dm-4 8 48 6297600 sdd 8 32 6297600 sdc 253 5 4208640 dm-5 253 6 4194304 dm-6 # blockdev --getsize /dev/sdc 12595200 # blockdev --getsize /dev/sdd 12595200 ]# multipath -ll mpath0 (30690a018c032c49fee6364140000d06d) dm-0 EQLOGIC,100E-00 [size=10G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 1:0:0:0 sdb 8:16 [active][ready] mpath-resize-test (36090a018c032e4801e8e341700008039) dm-5 EQLOGIC,100E-00 [size=4.0G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 3:0:0:0 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 2:0:0:0 sdd 8:48 [active][ready] # multipathd -k multipathd> del path sdc ok multipathd> add path sdc ok multipathd> del path sdd ok multipathd> add path sdd ok multipathd> resize map mpath-resize-test ok UPDATE 2009-03-23: With RHEL 5.3 GA it's only required to run the final "resize map" command with "multipathd -k". # multipath -ll mpath0 (30690a018c032c49fee6364140000d06d) dm-0 EQLOGIC,100E-00 [size=10G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 1:0:0:0 sdb 8:16 [active][ready] mpath-resize-test (36090a018c032e4801e8e341700008039) dm-5 EQLOGIC,100E-00 [size=6.0G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 3:0:0:0 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 2:0:0:0 sdd 8:48 [active][ready] # pvresize /dev/mapper/mpath-resize-test Physical volume "/dev/dm-5" changed 1 physical volume(s) resized / 0 physical volume(s) not resized # vgdisplay vgtest --- Volume group --- VG Name vgtest System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 6.00 GB PE Size 4.00 MB Total PE 1537 Alloc PE / Size 1024 / 4.00 GB Free PE / Size 513 / 2.00 GB VG UUID 9ER0Ph-u5gH-Y14g-w586-N31n-3L1X-qnPC11 # lvextend -L+2G /dev/vgtest/testvol Extending logical volume testvol to 6.00 GB Logical volume testvol successfully resized # resize2fs /dev/vgtest/testvol resize2fs 1.39 (29-May-2006) Filesystem at /dev/vgtest/testvol is mounted on /mnt/testvol; on-line resizing required Performing an on-line resize of /dev/vgtest/testvol to 1572864 (4k) blocks. The filesystem on /dev/vgtest/testvol is now 1572864 blocks long. # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg0-lvroot 7.6G 2.0G 5.2G 28% / /dev/mapper/mpath0p1 244M 15M 216M 7% /boot tmpfs 1.7G 0 1.7G 0% /dev/shm /dev/mapper/vgtest-testvol 6.0G 137M 5.5G 3% /mnt/testvol # touch /mnt/testvol/testfile # rm -rf /mnt/testvol/testfile # dmsetup info mpath-resize-test Name: mpath-resize-test State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 1 Major, minor: 253, 5 Number of targets: 1 UUID: mpath-36090a018c032e4801e8e341700008039 -------------------------------------------------------------------------------- So it seems to work for both ext3 directly on top of dm-mpath device, and for ext3 on top of LVM PV on top of dm-mpath device. --------------------------------------------------------------------------------