Logicalwebhost Cheatsheet

Linux & Open Source Cheatsheets & Howto's

Skip to: Content | Sidebar | Footer

RAID howto

In Linux you can either just use a hardware RAID, which makes all your disks look like a single one (or two) to you OS, or do a software RAID setup when you load the OS, which can look daunting if you’re new to it.

To do this you have to format each drive as “physical volume for RAID”, and then you create an MD device that will be the partition that goes on top of the RAID volumes you created. If you use RAID 1, that means you have a hard drive mirror, and so two hard drives RAID 1 would be the same capacity as one of the drives, but if it fails, you still have the other one.

RAID 5 and 6 offer more redundancy, but use more drives, like at least 3. RAID 6 means 2 drives can fail and still have your data, so it’s a little more robust than RAID 5, but takes more drives. On the other hand, big drives are a pain to try to recover, so it might be worth it. Here’s an example of what it looks like to set up RAID 6 in Debian, and then I put my /var directory, since I plan on that holding lots of stuff, and I boot the OS off a 30G SSD, since that’s fast:
raid6_debian

Once you create your RAID volumes and put stuff on them, you use mdadm tool to manage them, you start by getting the status like:

cat /proc/mdstat
  personalities : [raid6] [raid5] [raid4] 
  md0 : active raid6 sdb1[1] sdc1[2] sdd1[3]
      1464883200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/3] [_UUU]
      bitmap: 3/6 pages [12KB], 65536KB chunk
 
unused devices: <none>

This says I have a RAID 6 set up, but I know I *should* have really 4 drives in my RAID volume, but this only shows 3, so I check it out with:

mdadm -D /dev/md0
  /dev/md0:
        Version : 1.2
  Creation Time : Wed Dec  2 10:43:46 2015
     Raid Level : raid6
     Array Size : 1464883200 (1397.02 GiB 1500.04 GB)
  Used Dev Size : 732441600 (698.51 GiB 750.02 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent
  Intent Bitmap : Internal
    Update Time : Wed Jan 20 10:45:06 2016
          State : clean, degraded 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0
         Layout : left-symmetric
     Chunk Size : 512K
           Name : www:0  (local to host www)
           UUID : 574377b0:7ae2f061:ab417c11:357c8c53
         Events : 23258
    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1

You can tell by the list at the end that /dev/sda1 isn’t really hooked up (because it failed, then I replaced it), so now I have to re-add my /dev/sda1. Make TRIPLE SURE you are re-adding the right drive, or bad nasty things will happen :/ Once you do that, try to add it like:

mdadm --re-add /dev/md0 /dev/sda1
  mdadm: re-added /dev/sda1

Now you notice is shows up with the mdadm command like:

mdadm -D /dev/md0
  /dev/md0:
        Version : 1.2
  Creation Time : Wed Dec  2 10:43:46 2015
     Raid Level : raid6
     Array Size : 1464883200 (1397.02 GiB 1500.04 GB)
  Used Dev Size : 732441600 (698.51 GiB 750.02 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent
  Intent Bitmap : Internal
    Update Time : Wed Jan 20 10:49:38 2016
          State : clean, degraded, recovering 
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1
         Layout : left-symmetric
     Chunk Size : 512K
 Rebuild Status : 0% complete
           Name : www:0  (local to host www)
           UUID : 574377b0:7ae2f061:ab417c11:357c8c53
         Events : 23261
    Number   Major   Minor   RaidDevice State
       0       8        1        0      spare rebuilding   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1

See, now it says /dev/sda1 is rebuilding, you can check the process like:

cat /proc/mdstat
  Personalities : [raid6] [raid5] [raid4] 
  md0 : active raid6 sda1[0] sdb1[1] sdc1[2] sdd1[3]
      1464883200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/3] [_UUU]
      [>....................]  recovery =  0.2% (1978368/732441600) finish=178.4min speed=68219K/sec
      bitmap: 3/6 pages [12KB], 65536KB chunk
  unused devices: <none>

That means it is 0.2% done rebuilding. This process takes forever (it seems) if you have larger drives, or like hours at least. Keep checking back, and you should eventually see:

cat /proc/mdstat
  Personalities : [raid6] [raid5] [raid4] 
  md0 : active raid6 sda1[0] sdb1[1] sdc1[2] sdd1[3]
      1464883200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/6 pages [0KB], 65536KB chunk

Notice how it now shows my sda1 drive as part of the RAID 6 🙂

Also, mdadm will email you (if you have a default Postfix installed and set up your email address in /etc/aliases) if one of the drives does something weird, which is a super nice feature 🙂 Here’s how you set that up:

apt-get install postfix
  Select 'internet site'
vi /etc/aliases
  root: whateveruser
  whateveruser: enter@youremail.com
newaliases
vi /etc/mdadm/mdadm.conf
  MAILADDR root <-- make sure this it uncommented

to test a failed drive do:

mdadm --monitor --scan --test -1

if it worked, you should get an email summary from ‘mdadm monitoring’ of /proc/mdstat like:

This is an automatically generated mail message from mdadm
running on whateverhostname
A TestMessage event had been detected on md device /dev/md/1.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
  Personalities : [raid6] [raid5] [raid4] 
  md0 : active raid6 sda1[0] sdb1[1] sdc1[2] sdd1[3]
      1464883200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/6 pages [0KB], 65536KB chunk

If you want to manually remove a drive for some reason without breaking stuff, you can do the following, note: MAKE TRIPLE SURE YOU”RE REMOVING THE RIGHT DRIVE:

sudo mdadm --remove /dev/md0 /dev/sdb1

Configuring a RAID6 from scratch on Debian Jessie

This is not for the beginner. You have a very high chance of not getting it right and blowing up your ENTIRE data structure.

If you’re using large drives (over 2TB), you’ll have to use a different tool to format them, fdisk doesn’t work. Let’s start by seeing what is on your drives. We’ll need at least 3 drives for a RAID 6, so in my case I have my Operating System installed on a SSD called /dev/sda, so I’m starting my RAID volumes on /dev/sdb:

apt-get install parted gdisk mdadm
parted /dev/sdb
  (parted) p                                                                
  Model: ATA ST4000DM000-1F21 (scsi)
  Disk /dev/sdb: 4.00TB
  Sector size (logical/physical): 512B/4096B
  Partition Table: gpt
  Disk Flags: 
  Number  Start  End  Size  File system  Name  Flags
 
(parted)

This means you don’t have any partition table on there, or any partitions set up. So first you have to build a GPT partition table, then build a partition to fit on it like:

mklabel gpt
  Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
  Yes/No? yes
(parted)
(parted) unit TB

Okay, if there are no errors, you now have a usable disk, so now we’re going to create a single large RAID partition on the drive you just formatted like:

(parted) mkpart primary 0.00TB 4.00TB
(parted) p                                                                
  Model: ATA ST4000DM000-1F21 (scsi)
  Disk /dev/sdb: 4.00TB
  Sector size (logical/physical): 512B/4096B
  Partition Table: gpt
  Disk Flags: 
  Number  Start   End     Size    File system  Name     Flags
   1      0.00GB  4001GB  4001GB               primary

Now you have to create a RAID partition file system on it like:

(parted)set 1 raid on
(parted) p
  Model: ATA ST4000DM000-1F21 (scsi)
  Disk /dev/sdb: 4001GB
  Sector size (logical/physical): 512B/4096B
  Partition Table: gpt
  Disk Flags: 
  Number  Start   End     Size    File system  Name     Flags
   1      0.00GB  4001GB  4001GB               primary  raid
(parted) quit
Information: You may need to update /etc/fstab.

Now use fdisk to see if the OS sees it:

fdisk -l
  Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 4096 bytes
  I/O size (minimum/optimal): 4096 bytes / 4096 bytes
  Disklabel type: gpt
  Disk identifier: 29D2E5F9-7AFD-4DEA-8D15-9F3E3C8ED7C1
  Device     Start        End    Sectors  Size Type
  /dev/sdb1   2048 7814035455 7814033408  3.7T Linux RAID

Now you have to do the same formatting to the rest of your drives, in my case I have 5, so I’d do /dev/ sdc, sdd, sde, and sdf. Once you’re done with all that, you need mdadm, the RAID management tool, to build your RAID like:

mdadm --create --verbose /dev/md0 --level=6 --raid-devices=6 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: /dev/sdb1 appears to be part of a raid array:
       level=raid6 devices=4 ctime=Tue Jul 19 10:34:55 2016
mdadm: size set to 3906885632K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? yes
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
>: cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid6 sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      15627542528 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU]
      [>....................]  resync =  0.0% (1494792/3906885632) finish=827.3min speed=78673K/sec
      bitmap: 30/30 pages [120KB], 65536KB chunk
 
unused devices: <none>

This will take a very long time, like hours. You can format it in the meantime and use it. Also, the mdadm command will save it so it’s available the next time you reboot the machine. If you ever have to rebuild this array on a new machine, you can copy the /etc/mdadm/mdadm.conf file, and just run ‘mdadm –assemble –scan’.

To create the LVM on top of what you just build, use the command:

pvcreate -ff /dev/md0
  Really INITIALIZE physical volume "/dev/md0" of volume group "backupvol1" [y/n]? y
  WARNING: Forcing physical volume creation on /dev/md0 of volume group "backupvol1"
  Physical volume "/dev/md0" successfully created

Now you have to create your volume group like:

vgcreate raidvol1 /dev/md0
  Volume group "raidvol1" successfully created

Now you have to create the logical volume like:

lvcreate -l 100%FREE -n raidvol1lv raidvol1
  Logical volume "raidvol1lv" created

Now you format it like a normal filesystem:

mkfs -t ext4 /dev/raidvol1/raidvol1lv 
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 3906884608 4k blocks and 488361984 inodes
Filesystem UUID: d04aa16b-5f35-43a9-a0cd-35aa6fbfdc4f
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
        102400000, 214990848, 512000000, 550731776, 644972544, 1934917632, 
        2560000000, 3855122432
 
Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information:

Okay, now you want to mount it somewhere, so create a directory someplace and mount it:

mkdir /some/path/you/want
mount /dev/raidvol1/raidvol1lv /whatever/path/you/created/

Now it should be ready to use 🙂 If you want it to be seen the next time you reboot, add an entry to your /etc/fstab. To do that you should use the UUID of the volume, not just the /dev/raidvol1/raidvol1lv or some such. You find out what the UUID is by doing:

blkid -c /dev/null | grep raidvol
/dev/mapper/raidvol1-raidvol1lv: UUID="d04aa16b-5f35-43a9-a0cd-35aa6fbfdc4f" TYPE="ext4"

Now put that in the end of your /etc/fstab like:

vi /etc/fstab (add following line to end of it, but change as needed)
  #big raid6 volume used for whatever
  UUID=d04aa16b-5f35-43a9-a0cd-35aa6fbfdc4f /raid6volume  ext4    errors=remount-ro 0       1

Now if you reboot, you should see your raid volume, mounted wherever you wanted to mount it 🙂

RAID maintenance

Here are some commands to manage your RAID. First, here’s the status of your RAID you just created:

mdadm -D /dev/raidvol1/raidvol1lv
  /dev/raidvol1/raidvol1lv:
        Version : 1.2
  Creation Time : Thu Aug 25 11:37:17 2016
     Raid Level : raid6
     Array Size : 15627538432 (14903.58 GiB 16002.60 GB)
  Used Dev Size : 3906885632 (3725.90 GiB 4000.65 GB)
   Raid Devices : 6
  Total Devices : 6
    Persistence : Superblock is persistent
 
  Intent Bitmap : Internal
 
    Update Time : Fri Aug 26 14:53:52 2016
          State : active 
 Active Devices : 6
 Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0
 
         Layout : left-symmetric
     Chunk Size : 512K
 
           Name : debian:0  (local to host debian)
           UUID : 9b16b2c6:0d326920:377c3e6f:23b37baa
         Events : 17624
 
    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1
       4       8       65        4      active sync   /dev/sde1
       5       8       81        5      active sync   /dev/sdf1

If you want to check the status of one of the disks in the RAID, do:

mdadm -E /dev/sda1
  /dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9b16b2c6:0d326920:377c3e6f:23b37baa
           Name : debian:0  (local to host debian)
  Creation Time : Thu Aug 25 11:37:17 2016
     Raid Level : raid6
   Raid Devices : 6
 
 Avail Dev Size : 7813771264 (3725.90 GiB 4000.65 GB)
     Array Size : 15627542528 (14903.59 GiB 16002.60 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : active
    Device UUID : f28484a6:ac669b6c:61059466:6f8b45f4
 
Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Aug 26 14:58:07 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 468b744 - correct
         Events : 17676
 
         Layout : left-symmetric
     Chunk Size : 512K
 
   Device Role : Active device 0
   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

If a disk fails and you want to remove it do:

mdadm --remove /dev/md0 /dev/sda1

If you want to add a working one back in to begin the rebuild process, you have to format it first and make SURE the partitions match the other drives in the group. Once you do that, you can attempt to re-add it like:

mdadm --add /dev/md0 /dev/sda1

Sometimes Debian will decide it still needs to be marked as a spare and not used. If that’s the case, you’ll have to unmount and stop the RAID and try to re-assemble it like below, making SURE you use the –assume-clean switch or it will just rebuild the whole thing, which is very bad:

mdadm --stop /dev/md0
mdadm --create /dev/md0 --level=6 --raid-devices=4 --chunk=64 --name=RackStation:2 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 --assume-clean

Once you put it back in successfully, it will force mdadm to re-sync the RAID, which may take hours. You can monitor its progress by running:

watch -n .1 cat /proc/mdstat

If you want to see what physical volumes you’ve created do:

pvdisplay
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               raidvol1
  PV Size               14.55 TiB / not usable 4.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              3815317
  Free PE               0
  Allocated PE          3815317
  PV UUID               k8y1t4-dNGG-krdL-mXEp-20e0-GoHn-n0hRgx

To show what volume groups you have do:

vgdisplay
  --- Volume group ---
  VG Name               raidvol1
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               14.55 TiB
  PE Size               4.00 MiB
  Total PE              3815317
  Alloc PE / Size       3815317 / 14.55 TiB
  Free  PE / Size       0 / 0   
  VG UUID               YJDj53-vowl-GwEh-yYyt-UYJU-G4jG-uKWWKN

To show your logical volumes do:

lvdisplay
  --- Logical volume ---
  LV Path                /dev/raidvol1/raidvol1lv
  LV Name                raidvol1lv
  VG Name                raidvol1
  LV UUID                NxXvcw-pPE3-z4Cp-pGaW-tSLY-MEmP-zGYDFz
  LV Write Access        read/write
  LV Creation host, time debian, 2016-08-25 14:23:05 -0700
  LV Status              NOT available
  LV Size                14.55 TiB
  Current LE             3815317
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

To mount your volume groups:

vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "raidvol1" using metadata type lvm2
vgchange -ay raidvol1
  1 logical volume(s) in volume group "raidvol1" now active

Now you can mount them like:

mount /dev/raidvol1/raidvol1lv /wherever/you/want/tomount/it/
cd /wherever/you/just/mounted/it/

Now if it went well, you should have your raid up and working 🙂

: : :

There are some good RAID resources here.

If you have to recover an LVM, there’s a good reference here, you can do the same thing on Debian using the Debian LiveCD. Here’s an adaptation of what it says to do:

1. Get a live cd or USB, then boot to it rather than your host system you’re fixing.

2. Search for these tools: lvm2. If it’s not there, install it like:
# apt-get install lvm2

3. Use fdisk to figure out which drive(s) you’ll be recovering on. Note: this is not the disk you’re booting from, so MAKE SURE you’re about to work with the right disk.
# fdisk -lu

4. Once installed, run pvscan to scan all disks for physical volume. this to make sure your LVM harddisk is recognized by Debian.
# pvscan
PV /dev/sda2 VG VolGroup00 lvm2 [74.41 GB / 32.00 MB free]
Total: 1 [74.41 GB] / in use: 1 [74.41 GB] / in no VG: 0 [0 ]

5. After that run vgscan to scan disks for volume groups.
# vgscan
Reading all physical volumes. This may take a while…
Found volume group “VolGroup00” using metadata type lvm2

6. Activate all volume groups available.
# vgchange -a y
2 logical volume(s) in volume group “VolGroup00” now active

7. Run lvscan to scan all disks for logical volume. You can see partitions inside the hard disk now active.
# lvscan
ACTIVE ‘/dev/VolGroup00/LogVol00’ [72.44 GB] inherit
ACTIVE ‘/dev/VolGroup00/LogVol01’ [1.94 GB] inherit

8. Mount the partition to any directory you want, usually to /mnt
# mount /dev/VolGroup00/LogVol00 /mnt

9. You can access the partition in the /mnt directory and can backup your data using rsync

====

sfdisk -d /dev/sda | sfdisk /dev/sdh
sfdisk: Checking that no-one is using this disk right now …
sfdisk: OK

Disk /dev/sdh: 182397 cylinders, 255 heads, 63 sectors/track
Old situation:
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start End #cyls #blocks Id System
/dev/sdh1 0+ 182401- 182402- 1465136128 7 HPFS/NTFS/exFAT
/dev/sdh2 0 – 0 0 0 Empty
/dev/sdh3 0 – 0 0 0 Empty
/dev/sdh4 0 – 0 0 0 Empty
New situation:
Units: sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System
/dev/sdh1 * 2048 58593279 58591232 83 Linux
/dev/sdh2 58593280 64452607 5859328 fd Linux raid autodetect
/dev/sdh3 64452608 2930210815 2865758208 fd Linux raid autodetect
/dev/sdh4 0 – 0 0 Empty
sfdisk: Warning: partition 1 does not end at a cylinder boundary
sfdisk: Warning: partition 2 does not start at a cylinder boundary
sfdisk: Warning: partition 2 does not end at a cylinder boundary
sfdisk: Warning: partition 3 does not start at a cylinder boundary
sfdisk: Warning: partition 3 does not end at a cylinder boundary
Successfully wrote the new partition table

Re-reading the partition table …

sfdisk: If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)

mdadm –manage /dev/md0 –add /dev/sdh3
mdadm: added /dev/sdh3
root@backup2:/home/devr8ndom >: