Creating a test RAID5 + LVM storage pool

I'm currently planning an upgrade to my RHEL5 Linux storage server at home which will comprise three 1TB disks in a RAID5 configuration so there's at least some redundancy in case of a disk failure, and Logical Volume Manager (LVM) in case I decide to expand the pool down the track.

I thought it'd be a good idea to do a dry run using files instead of disks just to have a clear idea of what needs to be done.

There's a excellent walkthrough over at http://www.gagme.com/greg/linux/raid-lvm.php and that's what I based my configuration on, so check it out for a little more background info.

Create three zero-filled 1GB files:

for d in a b c; do
    dd if=/dev/zero of=/var/tmp/hd${d}.img bs=1024 count=1048576
done

Next, set up the loopback devices:

losetup /dev/loop0 /var/tmp/hda.img 
losetup /dev/loop1 /var/tmp/hdb.img 
losetup /dev/loop2 /var/tmp/hdc.img

Now we're going to use fdisk to create a 'Linux raid autodetect' partition on each 'disk':

fdisk /dev/loop0
fdisk /dev/loop1
fdisk /dev/loop2

The output should look like this, my commands in bold. Don't forget to do a partprobe once you're done.

[root@canard ~]# fdisk /dev/loop0
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-127, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-127, default 127): 
Using default value 127

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

partprobe

Now we're going to set up our RAID5 array on the md0 device with the 3 disks we just prepared:

mdadm --create /dev/md0 --level=5 --raid-devices=3 \
    /dev/loop0 /dev/loop1 /dev/loop2

And that's pretty much it for the RAID part. You can view that status of your md device using mdadm or viewing the proc entry, examples follow.

[root@canard ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 loop2[3] loop1[1] loop0[0]
      2047872 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
      [=========>...........]  recovery = 46.5% (476544/1023936) finish=4.3min speed=2112K/sec

[root@canard ~]# mdadm --detail /dev/md0 
/dev/md0:
        Version : 00.90.03
  Creation Time : Wed Jul 29 11:46:47 2009
     Raid Level : raid5
     Array Size : 2047872 (2000.21 MiB 2097.02 MB)
  Used Dev Size : 1023936 (1000.11 MiB 1048.51 MB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Jul 29 11:46:47 2009
          State : clean, degraded, recovering
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 53% complete

           UUID : 1744c4be:0566a9ee:8a056ff6:bd165eb4
         Events : 0.1

    Number   Major   Minor   RaidDevice State
       0       7        0        0      active sync   /dev/loop0
       1       7        1        1      active sync   /dev/loop1
       3       7        2        2      spare rebuilding   /dev/loop2

As you can see, md is in the process of building the array. Now it's time to configure the LVM side of things.

First create a PV (physical volume), then a VG (volume group) consisting of your PV, then a LV (logical volume) on top of your VG.

It's probably a good idea to check out the LVM docs prior to commencing this step to gain a good understanding of what it is and does.

You could just as easily leave things here and format the md0 device as ext3, mount it, and you'd be done. But we're not doing that just yet. So, onwards:

pvcreate /dev/md0
vgcreate vg_media /dev/md0

Now display some info on your VG - the output should look a little like this - pay special attention to the number of PE (physical extents) you have available:

[root@canard ~]# vgdisplay vg_media
  --- Volume group ---
  VG Name               vg_media
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.95 GB
  PE Size               4.00 MB
  Total PE              499 
  Alloc PE / Size       499 / 1.95 GB
  Free  PE / Size       0 / 0   
  VG UUID               2rsx3T-C1Lx-uMRf-kz1g-UyBP-tbTz-iC7K1f

OK so now we can go ahead and create the LV using the value for logical extents we obtained from the previous command:

lvcreate -l 499 vg_media -n lv_media

And again, display some information:

[root@canard ~]# lvdisplay /dev/vg_media/lv_media
  --- Logical volume ---
  LV Name                /dev/vg_media/lv_media
  VG Name                vg_media
  LV UUID                iS01Vt-Hyo7-GIiA-Sxlu-rc5J-bX5g-Trd9Ln
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.95 GB
  Current LE             499
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

We're nearly done! All that's left to do is format and mount the LV:

mkfs.ext3 /dev/vg_media/lv_media
mkdir /mnt/lvtest
mount /dev/vg_media/lv_media /mnt/lvtest

You'll notice that df reports the parity space as being used already, so out of the 3GB we should have around 2GB available for use:

[root@canard ~]# df -h /mnt/lvtest/
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_media-lv_media
                      2.0G   35M  1.8G   2% /mnt/lvtest

Next I'll be failing a disk and expanding the LV to incorporate a second RAID5 array. As ever, stay tuned.