In Linux you can either just use a hardware RAID, which makes all your disks look like a single one (or two) to you OS, or do a software RAID setup when you load the OS, which can look daunting if you’re new to it.
To do this you have to format each drive as “physical volume for RAID”, and then you create an MD device that will be the partition that goes on top of the RAID volumes you created. If you use RAID 1, that means you have a hard drive mirror, and so two hard drives RAID 1 would be the same capacity as one of the drives, but if it fails, you still have the other one.
RAID 5 and 6 offer more redundancy, but use more drives, like at least 3. RAID 6 means 2 drives can fail and still have your data, so it’s a little more robust than RAID 5, but takes more drives. On the other hand, big drives are a pain to try to recover, so it might be worth it. Here’s an example of what it looks like to set up RAID 6 in Debian, and then I put my /var directory, since I plan on that holding lots of stuff, and I boot the OS off a 30G SSD, since that’s fast:
Once you create your RAID volumes and put stuff on them, you use mdadm tool to manage them, you start by getting the status like:
cat /proc/mdstat personalities : [raid6] [raid5] [raid4] md0 : active raid6 sdb1 sdc1 sdd1 1464883200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/3] [_UUU] bitmap: 3/6 pages [12KB], 65536KB chunk unused devices: <none>
This says I have a RAID 6 set up, but I know I *should* have really 4 drives in my RAID volume, but this only shows 3, so I check it out with:
mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Wed Dec 2 10:43:46 2015 Raid Level : raid6 Array Size : 1464883200 (1397.02 GiB 1500.04 GB) Used Dev Size : 732441600 (698.51 GiB 750.02 GB) Raid Devices : 4 Total Devices : 3 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Wed Jan 20 10:45:06 2016 State : clean, degraded Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : www:0 (local to host www) UUID : 574377b0:7ae2f061:ab417c11:357c8c53 Events : 23258 Number Major Minor RaidDevice State 0 0 0 0 removed 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1
You can tell by the list at the end that /dev/sda1 isn’t really hooked up (because it failed, then I replaced it), so now I have to re-add my /dev/sda1. Make TRIPLE SURE you are re-adding the right drive, or bad nasty things will happen :/ Once you do that, try to add it like:
mdadm --re-add /dev/md0 /dev/sda1 mdadm: re-added /dev/sda1
Now you notice is shows up with the mdadm command like:
mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Wed Dec 2 10:43:46 2015 Raid Level : raid6 Array Size : 1464883200 (1397.02 GiB 1500.04 GB) Used Dev Size : 732441600 (698.51 GiB 750.02 GB) Raid Devices : 4 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Wed Jan 20 10:49:38 2016 State : clean, degraded, recovering Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 512K Rebuild Status : 0% complete Name : www:0 (local to host www) UUID : 574377b0:7ae2f061:ab417c11:357c8c53 Events : 23261 Number Major Minor RaidDevice State 0 8 1 0 spare rebuilding /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1
See, now it says /dev/sda1 is rebuilding, you can check the process like:
cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid6 sda1 sdb1 sdc1 sdd1 1464883200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/3] [_UUU] [>....................] recovery = 0.2% (1978368/732441600) finish=178.4min speed=68219K/sec bitmap: 3/6 pages [12KB], 65536KB chunk unused devices: <none>
That means it is 0.2% done rebuilding. This process takes forever (it seems) if you have larger drives, or like hours at least. Keep checking back, and you should eventually see:
cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid6 sda1 sdb1 sdc1 sdd1 1464883200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU] bitmap: 0/6 pages [0KB], 65536KB chunk
Notice how it now shows my sda1 drive as part of the RAID 6 🙂
Also, mdadm will email you (if you have a default Postfix installed and set up your email address in /etc/aliases) if one of the drives does something weird, which is a super nice feature 🙂 Here’s how you set that up:
apt-get install postfix Select 'internet site' vi /etc/aliases root: whateveruser whateveruser: firstname.lastname@example.org newaliases vi /etc/mdadm/mdadm.conf MAILADDR root <-- make sure this it uncommented
to test a failed drive do:
mdadm --monitor --scan --test -1
if it worked, you should get an email summary from ‘mdadm monitoring’ of /proc/mdstat like:
This is an automatically generated mail message from mdadm running on whateverhostname A TestMessage event had been detected on md device /dev/md/1. Faithfully yours, etc. P.S. The /proc/mdstat file currently contains the following: Personalities : [raid6] [raid5] [raid4] md0 : active raid6 sda1 sdb1 sdc1 sdd1 1464883200 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU] bitmap: 0/6 pages [0KB], 65536KB chunk
If you want to manually remove a drive for some reason without breaking stuff, you can do the following, note: MAKE TRIPLE SURE YOU”RE REMOVING THE RIGHT DRIVE:
sudo mdadm --remove /dev/md0 /dev/sdb1
Configuring a RAID6 from scratch on Debian Jessie
This is not for the beginner. You have a very high chance of not getting it right and blowing up your ENTIRE data structure.
If you’re using large drives (over 2TB), you’ll have to use a different tool to format them, fdisk doesn’t work. Let’s start by seeing what is on your drives. We’ll need at least 3 drives for a RAID 6, so in my case I have my Operating System installed on a SSD called /dev/sda, so I’m starting my RAID volumes on /dev/sdb:
apt-get install parted gdisk mdadm parted /dev/sdb (parted) p Model: ATA ST4000DM000-1F21 (scsi) Disk /dev/sdb: 4.00TB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags (parted)
This means you don’t have any partition table on there, or any partitions set up. So first you have to build a GPT partition table, then build a partition to fit on it like:
mklabel gpt Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue? Yes/No? yes (parted) (parted) unit TB
Okay, if there are no errors, you now have a usable disk, so now we’re going to create a single large RAID partition on the drive you just formatted like:
(parted) mkpart primary 0.00TB 4.00TB (parted) p Model: ATA ST4000DM000-1F21 (scsi) Disk /dev/sdb: 4.00TB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 0.00GB 4001GB 4001GB primary
Now you have to create a RAID partition file system on it like:
(parted)set 1 raid on (parted) p Model: ATA ST4000DM000-1F21 (scsi) Disk /dev/sdb: 4001GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 0.00GB 4001GB 4001GB primary raid (parted) quit Information: You may need to update /etc/fstab.
Now use fdisk to see if the OS sees it:
fdisk -l Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 29D2E5F9-7AFD-4DEA-8D15-9F3E3C8ED7C1 Device Start End Sectors Size Type /dev/sdb1 2048 7814035455 7814033408 3.7T Linux RAID
Now you have to do the same formatting to the rest of your drives, in my case I have 5, so I’d do /dev/ sdc, sdd, sde, and sdf. Once you’re done with all that, you need mdadm, the RAID management tool, to build your RAID like:
mdadm --create --verbose /dev/md0 --level=6 --raid-devices=6 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 512K mdadm: /dev/sdb1 appears to be part of a raid array: level=raid6 devices=4 ctime=Tue Jul 19 10:34:55 2016 mdadm: size set to 3906885632K mdadm: automatically enabling write-intent bitmap on large array Continue creating array? yes mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. >: cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid6 sdf1 sde1 sdd1 sdc1 sdb1 sda1 15627542528 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/6] [UUUUUU] [>....................] resync = 0.0% (1494792/3906885632) finish=827.3min speed=78673K/sec bitmap: 30/30 pages [120KB], 65536KB chunk unused devices: <none>
This will take a very long time, like hours. You can format it in the meantime and use it. Also, the mdadm command will save it so it’s available the next time you reboot the machine. If you ever have to rebuild this array on a new machine, you can copy the /etc/mdadm/mdadm.conf file, and just run ‘mdadm –assemble –scan’.
To create the LVM on top of what you just build, use the command:
pvcreate -ff /dev/md0 Really INITIALIZE physical volume "/dev/md0" of volume group "backupvol1" [y/n]? y WARNING: Forcing physical volume creation on /dev/md0 of volume group "backupvol1" Physical volume "/dev/md0" successfully created
Now you have to create your volume group like:
vgcreate raidvol1 /dev/md0 Volume group "raidvol1" successfully created
Now you have to create the logical volume like:
lvcreate -l 100%FREE -n raidvol1lv raidvol1 Logical volume "raidvol1lv" created
Now you format it like a normal filesystem:
mkfs -t ext4 /dev/raidvol1/raidvol1lv mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 3906884608 4k blocks and 488361984 inodes Filesystem UUID: d04aa16b-5f35-43a9-a0cd-35aa6fbfdc4f Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848, 512000000, 550731776, 644972544, 1934917632, 2560000000, 3855122432 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information:
Okay, now you want to mount it somewhere, so create a directory someplace and mount it:
mkdir /some/path/you/want mount /dev/raidvol1/raidvol1lv /whatever/path/you/created/
Now it should be ready to use 🙂 If you want it to be seen the next time you reboot, add an entry to your /etc/fstab. To do that you should use the UUID of the volume, not just the /dev/raidvol1/raidvol1lv or some such. You find out what the UUID is by doing:
blkid -c /dev/null | grep raidvol /dev/mapper/raidvol1-raidvol1lv: UUID="d04aa16b-5f35-43a9-a0cd-35aa6fbfdc4f" TYPE="ext4"
Now put that in the end of your /etc/fstab like:
vi /etc/fstab (add following line to end of it, but change as needed) #big raid6 volume used for whatever UUID=d04aa16b-5f35-43a9-a0cd-35aa6fbfdc4f /raid6volume ext4 errors=remount-ro 0 1
Now if you reboot, you should see your raid volume, mounted wherever you wanted to mount it 🙂
Here are some commands to manage your RAID. First, here’s the status of your RAID you just created:
mdadm -D /dev/raidvol1/raidvol1lv /dev/raidvol1/raidvol1lv: Version : 1.2 Creation Time : Thu Aug 25 11:37:17 2016 Raid Level : raid6 Array Size : 15627538432 (14903.58 GiB 16002.60 GB) Used Dev Size : 3906885632 (3725.90 GiB 4000.65 GB) Raid Devices : 6 Total Devices : 6 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 26 14:53:52 2016 State : active Active Devices : 6 Working Devices : 6 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : debian:0 (local to host debian) UUID : 9b16b2c6:0d326920:377c3e6f:23b37baa Events : 17624 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1 4 8 65 4 active sync /dev/sde1 5 8 81 5 active sync /dev/sdf1
If you want to check the status of one of the disks in the RAID, do:
mdadm -E /dev/sda1 /dev/sda1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 9b16b2c6:0d326920:377c3e6f:23b37baa Name : debian:0 (local to host debian) Creation Time : Thu Aug 25 11:37:17 2016 Raid Level : raid6 Raid Devices : 6 Avail Dev Size : 7813771264 (3725.90 GiB 4000.65 GB) Array Size : 15627542528 (14903.59 GiB 16002.60 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262056 sectors, after=0 sectors State : active Device UUID : f28484a6:ac669b6c:61059466:6f8b45f4 Internal Bitmap : 8 sectors from superblock Update Time : Fri Aug 26 14:58:07 2016 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 468b744 - correct Events : 17676 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
If a disk fails and you want to remove it do:
mdadm --remove /dev/md0 /dev/sda1
If you want to add a working one back in to begin the rebuild process, you have to format it first and make SURE the partitions match the other drives in the group. Once you do that, you can attempt to re-add it like:
mdadm --add /dev/md0 /dev/sda1
Sometimes Debian will decide it still needs to be marked as a spare and not used. If that’s the case, you’ll have to unmount and stop the RAID and try to re-assemble it like below, making SURE you use the –assume-clean switch or it will just rebuild the whole thing, which is very bad:
mdadm --stop /dev/md0 mdadm --create /dev/md0 --level=6 --raid-devices=4 --chunk=64 --name=RackStation:2 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 --assume-clean
Once you put it back in successfully, it will force mdadm to re-sync the RAID, which may take hours. You can monitor its progress by running:
watch -n .1 cat /proc/mdstat
If you want to see what physical volumes you’ve created do:
pvdisplay --- Physical volume --- PV Name /dev/md0 VG Name raidvol1 PV Size 14.55 TiB / not usable 4.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 3815317 Free PE 0 Allocated PE 3815317 PV UUID k8y1t4-dNGG-krdL-mXEp-20e0-GoHn-n0hRgx
To show what volume groups you have do:
vgdisplay --- Volume group --- VG Name raidvol1 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 14.55 TiB PE Size 4.00 MiB Total PE 3815317 Alloc PE / Size 3815317 / 14.55 TiB Free PE / Size 0 / 0 VG UUID YJDj53-vowl-GwEh-yYyt-UYJU-G4jG-uKWWKN
To show your logical volumes do:
lvdisplay --- Logical volume --- LV Path /dev/raidvol1/raidvol1lv LV Name raidvol1lv VG Name raidvol1 LV UUID NxXvcw-pPE3-z4Cp-pGaW-tSLY-MEmP-zGYDFz LV Write Access read/write LV Creation host, time debian, 2016-08-25 14:23:05 -0700 LV Status NOT available LV Size 14.55 TiB Current LE 3815317 Segments 1 Allocation inherit Read ahead sectors auto
To mount your volume groups:
vgscan Reading all physical volumes. This may take a while... Found volume group "raidvol1" using metadata type lvm2 vgchange -ay raidvol1 1 logical volume(s) in volume group "raidvol1" now active
Now you can mount them like:
mount /dev/raidvol1/raidvol1lv /wherever/you/want/tomount/it/ cd /wherever/you/just/mounted/it/
Now if it went well, you should have your raid up and working 🙂
: : :
There are some good RAID resources here.
If you have to recover an LVM, there’s a good reference here, you can do the same thing on Debian using the Debian LiveCD. Here’s an adaptation of what it says to do:
1. Get a live cd or USB, then boot to it rather than your host system you’re fixing.
2. Search for these tools: lvm2. If it’s not there, install it like:
# apt-get install lvm2
3. Use fdisk to figure out which drive(s) you’ll be recovering on. Note: this is not the disk you’re booting from, so MAKE SURE you’re about to work with the right disk.
# fdisk -lu
4. Once installed, run pvscan to scan all disks for physical volume. this to make sure your LVM harddisk is recognized by Debian.
PV /dev/sda2 VG VolGroup00 lvm2 [74.41 GB / 32.00 MB free]
Total: 1 [74.41 GB] / in use: 1 [74.41 GB] / in no VG: 0 [0 ]
5. After that run vgscan to scan disks for volume groups.
Reading all physical volumes. This may take a while…
Found volume group “VolGroup00” using metadata type lvm2
6. Activate all volume groups available.
# vgchange -a y
2 logical volume(s) in volume group “VolGroup00” now active
7. Run lvscan to scan all disks for logical volume. You can see partitions inside the hard disk now active.
ACTIVE ‘/dev/VolGroup00/LogVol00’ [72.44 GB] inherit
ACTIVE ‘/dev/VolGroup00/LogVol01’ [1.94 GB] inherit
8. Mount the partition to any directory you want, usually to /mnt
# mount /dev/VolGroup00/LogVol00 /mnt
9. You can access the partition in the /mnt directory and can backup your data using rsync