Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

The Linux Logical Volume Manager (LVM)

General concepts of logical volume manager (LVM) stems from the desire to be able to create filesystems that span several physical disks as well as change the size of existing partitions of the fly. The Linux LVM implementation is similar to the HP-UX LVM implementation. It was originally written in 1998 by Heinz Mauelshagen. A good introduction to the basic concepts can be found in Wikipedia articles  Logical volume management   and  Logical Volume Manager (Linux)

Some code was donated by IBM [IBM pitches its open source side]. It is unclear is it is still used. See Enterprise Volume Management System - Wikipedia

IBM has donated technology, code and skills to the Linux community, Kloeckner said, citing the company's donation of the Logical Volume Manager and its Journaling File System.

Matthew O'Keefe who from 1990 to May 2000, taught and performed research in storage systems and parallel simulation software as a professor of electrical and computer engineering at the University of Minnesota founded Sistina Software in May of 2000 to develop storage infrastructure software for Linux, including the Linux Logical Volume Manager (LVM).  They created LVM2.  Sistina was acquired by Red Hat in December 2003.

LVM2 is identical in Red Hat and Suse although it have different GUI interface for managing volumes. The installers for both Red Hat and Suse  are LVM-aware.

Although Linux volume manager works OK and is pretty reliable, documentation sucks badly for a commercial product. The most readable documentation that I have found is the article by Klaus Heinrich Kiwi Logical volume management published at IBM Developer Works on September 11, 2007. 

Moreover in RHEL 4 GUI interface is almost unusable as the left pane cannot be enlarged.

The LVM hierarchy includes  Physical Volume (PV) (typically a hard disk or partition, though it may well just be a device that 'looks' like a hard disk e.g. a  RAID device).  Volume Group (VG)  (the new virtual disk that can contain several physical disks) and Logical Volumes (LV) -- the equivalent of a disk partition in a non-LVM system.  The Volume Group is the highest level abstraction used within the LVM.

    hda1   hdc1      (PV:s on partitions or whole disks)                        
       \   /                                                                    
        \ /                                                                     
       diskvg        (VG)                                                       
       /  |  \                                                                  
      /   |   \                                                                 
  usrlv rootlv varlv (LV:s)
    |      |     |                                                              
 ext2  reiserfs  xfs (filesystems)

The lowest level in the LVM storage hierarchy is the Physical Volume (PV). A PV is a single device or partition and is created with the command: pvcreate device. This step initializes a partition for later use.  During this step each physical volume is divided chunks of data, known as physical extents, these extents have the same size as the logical extents for the volume group.

Multiple Physical Volumes (initialized partitions) are merged into a Volume Group (VG). This is done with the command: vgcreate volume_name device {device}.  This step also registers  volume_name in the LVM kernel module and therefore it is made accessible to the kernel I/O layer. For example:

vgcreate test-volume /dev/hda2 /dev/hda10

A Volume Group is pool from which Logical Volumes (LV) can be allocated. LV is the equivalent of a disk partition in a non-LVM system. The LV is visible as a standard block device; as such the LV can contain a file system (eg. /home).  Creating an LV is done with  lvcreate command

LVM Tools

Operations on Logical Volumes

Among typical operations:

A Walkthrough of the LVM for Linux

Adding a disk to the Volume Group

Next, we'll add /dev/hda6 to the Volume Group. Just type vgextend vg01 /dev/hda6 and you're done! You can check this out by using vgdisplay -v vg01. Note that there are now a lot more PEs available!

Moving Creating a striped Logical Volume

Note that LVM created your whole Logical Volume on one Physical Volume within the Volume Group. You can also stripe an LV across two Physical Volumes with the -i flag in lvcreate. We'll create a new LV, lv02, striped across hda5 and hda6. Type lvcreate -l4 -nlv02 -i2 vg01 /dev/hda5 /dev/hda6. Specifying the PV on the command line tells LVM which PEs to use, while the -i2 command tells it to stripe it across the two.

You now have an LV striped across two PVs!

Moving data within a Volume Group

Up to now, PEs and LEs were pretty much interchangable. They are the same size and are mapped automatically by LVM. This does not have to be the case, though. In fact, you can move an entire LV from one PV to another, even while the disk is mounted and in use! This will impact your performance, but it can prove useful.

Let's move lv01 to hda6 from hda5. Type pvmove -n/dev/vg01/lv01 /dev/hda5 /dev/hda6. This will move all LEs used by lv01 mapped to PEs on /dev/hda5 to new PEs on /dev/hda6. Effectively, this migrates data from hda5 to hda6. It takes a while, but when it's done, take a look with lvdisplay -v /dev/vg01/lv01 and notice that it now resides entirely on /dev/hda6!

Removing a Logical Volume from a Volume Group

Let's say we no longer need lv02. We can remove it and place its PEs back in the empty pool for the Volume Group. First, unmounting its filesystem. Next, deactivate it with lvchange -a n /dev/vg01/lv02. Finally, delete it by typing lvremove /dev/vg01/lv02. Look at the Volume Group and notice that the PEs are now unused.

Removing a disk from the Volume Group

You can also remove a disk from a volume group. We aren't using hda5 anymore, so we can remove it from the Volume Group. Just type vgreduce vg01 /dev/hda5 and it's gone!

A file system on logical volume may be extended. Also more space may be added to a VG by adding new partitions or devices  with the command: vgextend. For example:

lvextend -L +4G /dev/VolGroup00/LogVol04

The command pvmove can be used in several ways to move any LV elsewhere. There are also many more commands to rename, remove, split, merge, activate, deactivate and get extended information about current PV's, VG's and LV's.

Here is a typical du map of a server with volume manager installed. As you can see all partitions except /boot partition are referred vi path /dev/mapper/VolGroup00-LogVolxx where xx is two digit number:

				
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       4128448    316304   3602432   9% /
/dev/sda3               194449     22382    162027  13% /boot
none                   2020484         0   2020484   0% /dev/shm
/dev/mapper/VolGroup00-LogVol05
                       4128448     42012   3876724   2% /home
/dev/mapper/VolGroup00-LogVol03
                       4128448     41640   3877096   2% /tmp
/dev/mapper/VolGroup00-LogVol02
                       8256952   3189944   4647580  41% /usr
/dev/mapper/VolGroup00-LogVol04
                       8256952    174232   7663344   3% /var
/dev/hde                594366    594366         0 100% /media/cdrecorder
				

Resiliency to renumbering of physical hard disks

LVM identifies PVs by UUID, not by device name.

Each disk (PV) is labeled with a UUID, which uniquely identifies it to the system. 'vgscan' identifies this after a new disk is added that changes your drive numbering. Most distros run vgscan in the lvm startup scripts to cope with this on reboot after a hardware addition. If you're doing a hot-add, you'll have to run this by hand I think. On the other hand, if your vg is activated and being used, the renumbering should not affect it at all. It's only the activation that needs the identifier, and the worst case scenario is that the activation will fail without a vgscan with a complaint about a missing PV.

The failure or removal of a drive that LVM is currently using will cause problems with current use and future activations of the VG that was using it.

How to get information about free space

vgdisplay shows logical volumes one by one and provides the information about free disk space on each:

vgdisplay volume_group_one | grep "Total PE"

How to create a new volume

# vgcreate test-volume /dev/hda2 /dev/hda10
  Volume group "test-volume" successfully created

How to create and mount a partition

1. Create the partition with lvcreate

   # lvcreate -L 5G -n data vg02
     Logical volume "data" created
    

2. Format partition

   # mkfs -t ext3 /dev/vg02/data
    

3. Make mount point and mount it

   # mkdir /data
   # mount /dev/vg02/data /data/

4. Check results

   # df -h /data
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/mapper/test--volume-data
                         50.0G   33M  5.0G   1% /data
    

5. Add it to /etc/fstab

How to extend the partition

If one wishes to use all the free physical extents on the volume group one can achieve this using the lvm lvextend command :

lvm lvextend -L +4G /dev/VolGroup00/LogVol04 # extend /var

ext2online /dev/VolGroup00/LogVol04

# lvm lvextend -l+7153 /dev/TestVG/TestLV
  Extending logical volume TestLV to 30.28 GB
  Logical volume TestLV successfully resized

 "lvextend -L +54 /dev/vg01/lvol10 /dev/sdk3" tries to extend the size of that logical volume by 54MB on physical volume /dev/sdk3. This is only possible if /dev/sdk3 is a member of volume group vg01.

Then the pvcreate command is used to create the new physical volume using the new partition, and the pvs again to verify the new physical volume. See  redhat.com Knowledgebase

How to resize the file system with ext2online

After extending the volume group and the logical volume, it is now possible to resize the file system on the fly. This is done using ext2online. First I verify the file system size, perform the resize, and then verify the size again:

# df -h /mnt/test
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/TestVG-TestLV
                      2.3G   36M  2.2G   2% /mnt/test

# ext2online /dev/TestVG/TestLV
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b

# df -h /mnt/test
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/TestVG-TestLV
                       30G   39M   29G   1% /mnt/test

Old News ;-)

[Nov 12, 2008] /dev/dm-0 - LinuxQuestions.org

fdisk -l output in case you are using LVM contains many messages like Disk /dev/dm-0 doesn't contain a valid partition table

This has been very helpful to me. I found this thread by Goggle on
dm-0 because I also got the no partition table error message.

Here is what I think:

When the programs fdisk and sfdisk are run with the option -l and no argument, e.g.
# /sbin/fdisk -l

they look for all devices that can have cylinders, heads, sectors, etc.
If they find such a device, they output that information to standard
output and they output the partition table to standard output. If there is
no partition table, they have an error message (also standard output).
One can see this by piping to 'less', e.g.
# /sbin/fdisk -l | less

/dev/dm-0 ... /dev/dm3 on my fedora C5 system seem to be device mappers
associated with LVM.

RAID might also require device mappers.

[Nov 11, 2008] EXT3 filesystem recovery in LVM2

EXT3 filesystem recovery in LVM2 
--------------------------------------------------------------------------------
This is the bugzilla bug I started on the fedora buzilla:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=142737
--------------------------------------------------------------------------------
Very good idea to do something like the following, so that you have
an copy of the partition you're trying to recover, in case something
bad happens:
dd if=/dev/hda2 bs=1024k conv=noerror,sync,notrunc | reblock -t 65536 30 |
ssh remote.host.uci.edu 'cat > /recovery/damaged-lvm2-ext3'
--------------------------------------------------------------------------------
e2salvage died with "Terminated".  I assume it OOM'd.
--------------------------------------------------------------------------------
e2extract gave a huge list of 0 length files.  Doesn't seem right,
and it was taking forever, so I decided to move on to other methods.
But does anyone know if this is normal behavior for e2extract on an ext3?
--------------------------------------------------------------------------------
I wrote a small program that searches for ext3 magic numbers.  It's
finding many, EG 438, 30438, 63e438 and so on (hex).  The question is,
how do I convert from that to an fsck -b number?
--------------------------------------------------------------------------------
Running the same program on a known-good ext3, the first offset was the
same, but others were different.  However, they all ended in hex 38...
--------------------------------------------------------------------------------
I'm now running an "fsck -vn -b" with the -b argument ranging from 0 to
999999.  I'm hoping this will locate a suitable -b for me via brute force.
--------------------------------------------------------------------------------
Sent a post to gmane.linux.kernel 2004-12-16 
--------------------------------------------------------------------------------
Robin Green  very helpfully provided the
following instructions, which appear to be getting somewhere:

1) Note down what the root= device is that appears on the kernel
command line (this can be found by going to boot from hard drive and
then examining
the kernel command line in grub, or by looking in /boot/grub/grub.conf )

2) Be booted from rescue disk

3) Sanity check: ensure that the nodes /dev/hda, /dev/hda2 etc. exist

4) Start up LVM2 (assuming it is not already started by the rescue disk!) by
typing:

  lvm vgchange --ignorelockingfailure -P -a y

Looking at my initrd script, it doesn't seem necessary to run any other
commands
to get LVM2 volumes activated - that's it.

5) Find out which major/minor number the root device is. This is the
slightly tricky
bit. You may have to use trial-and-error. In my case, I guessed right
first time:
(no comments about my odd hardware setup please ;)

[root@localhost t]# ls /sys/block
dm-0  dm-2  hdd    loop1  loop3  loop5  loop7  ram0  ram10  ram12  ram14
ram2  ram4  ram6  ram8
dm-1  hdc   loop0  loop2  loop4  loop6  md0    ram1  ram11  ram13  ram15
ram3  ram5  ram7  ram9
[root@localhost t]# cat /sys/block/dm-0/dev
253:0
[root@localhost t]# devmap_name 253 0
Volume01-LogVol02

In the first command, I listed the block devices known to the kernel. dm-*
are the LVM
devices (on my 2.6.9 kernel, anyway). In the second command, I found
out the major:minor
numbers of /dev/dm-0. In the third command, I used devmap_name to check
that the device
mapper name of node with major 253 and minor 0, is the same as the name
of the root device
from my kernel command line (cf. step 1). Apart from a slight punctuation
difference,
it is the same, therefore I have found the root device.

I'm not sure if FC3 includes the devmap_name command. According to
fr2.rpmfind.net, it doesn't.
But you don't really need it, you can just try all the LVM devices in
turn until you find
your root device. Or, I can email you a statically-linked binary of it
if you want.

6) Create the /dev node for the root filesystem if it doesn't already
exist, e.g.:

  mknod /dev/dm-0 b 253 0

using the major-minor numbers found in step 5.

Please note that for the purpose of _rescue_, the node doesn't actually
have to be under
/dev (so /dev doesn't have to be writeable) and its name does not
matter. It just needs
to exist somewhere on a filesystem, and you have to refer to it in the
next command.

7) Do what you want to the root filesystem, e.g.:

  fsck /dev/dm-0
  mount /dev/dm-0 /where/ever

As you probably know, the fsck might actually work, because a fsck
can sometimes
correct filesystem errors that the kernel filesystem modules cannot.

8) If the fsck doesn't work, look in the output of fsck and in dmesg
for signs of
physical drive errors. If you find them, (a) think about calling a
data recovery
specialist, (b) do NOT use the drive!
--------------------------------------------------------------------------------
On FC3's rescue disk, what I actually did was:

1) Do startup network interfaces
2) Don't try to automatically mount the filesystems - not even readonly
3) lvm vgchange --ignorelockingfailure -P -a y
4) fdisk -l, and guess which partition is which based on size: the small
one was /boot, and the large one was /
5) mkdir /mnt/boot
6) mount /dev/hda1 /mnt/boot
7) Look up the device node for the root filesystem in /mnt/boot/grub/grub.conf
8) A first tentative step, to see if things are working: fsck -n
/dev/VolGroup00/LogVol00
9) Dive in: fsck -f -y /dev/VolGroup00/LogVol00
10) Wait a while...  Be patient.  Don't interrupt it
11) Reboot
--------------------------------------------------------------------------------
Are these lvm1 or lvm2?

lvmdiskscan -v
vgchange -ay
vgscan -P
vgchange -ay -P
--------------------------------------------------------------------------------
jeeves:~# lvm version
  LVM version:     2.01.04 (2005-02-09)
  Library version: 1.01.00-ioctl (2005-01-17)
  Driver version:  4.1.0
--------------------------------------------------------------------------------
I think you are making a potentially very dangerous mistake!

Type 8e is a partition type. You don't want to use resize2fs on the PARTITION,
which is not an ext2 partition, but an lvm partition. You want
to resize the filesystem on the logical VOLUME.

And yes, resize2fs is appropriate for logical volumes. But resize the VOLUME
(e.g. /dev/VolGroup00/LogVol00), not the partition or volume group.

On Fri, Mar 04, 2005 at 06:35:31PM +0000, Robert Buick wrote:
> I'm using type 8e, does anyone happen to know if resize2fs is
> appropriate for this type; the man page only mentions type2.
--------------------------------------------------------------------------------
A method of hunting for two text strings in a raw disk, after files
have been deleted.  The data blocks of the disk are read once, but
grep'd twice.

seki-root> reblock -e 75216016 $(expr 1024 \* 1024) 300 <
/dev/mapper/VolGroup00-LogVol00 | mtee 'egrep --binary-files=text -i -B
1000 -A 1000 dptutil > dptutil-hits' 'egrep --binary-files=text -i
-B 1000 -A 1000 dptmgr > dptmgr-hits'
stdin seems seekable, but file length is 0 - no exact percentages
Estimated filetransfer size is 77021200384 bytes
Estimated percentages will only be as accurate as your size estimate
Creating 2 pipes
popening egrep --binary-files=text -i -B 1000 -A 1000 dptutil > dptutil-hits
popening egrep --binary-files=text -i -B 1000 -A 1000 dptmgr > dptmgr-hits
(estimate: 0.1%  0s 56m 11h) Kbytes: 106496.0  Mbits/s: 13.6  Gbytes/hr:
6.0  min: 1.0
(estimate: 0.2%  9s 12m 12h) Kbytes: 214016.0  Mbits/s: 13.3  Gbytes/hr:
5.8  min: 2.0
(estimate: 0.3%  58s 58m 11h) Kbytes: 257024.0  Mbits/s: 13.5  Gbytes/hr:
5.9  min: 2.4
...

references:
http://stromberg.dnsalias.org/~strombrg/reblock.html
http://stromberg.dnsalias.org/~strombrg/mtee.html
egrep --help
--------------------------------------------------------------------------------
Performing the above reblock | mtee, my fedora core 3 system got -very-
slow.  If I were to suspend the pipeline above, performance would be
great.  If I resumed it, very quickly, performance would be bad again.
This command seems to have left my sytem a little bit jerky, but it's
-far- more usable now, despite the pipeline above still pounding the
SATA drive my home directory is on.

seki-root> echo deadline > scheduler 
Wed Mar 09 17:56:58

seki-root> cat scheduler 
noop anticipatory [deadline] cfq 
Wed Mar 09 17:57:00

seki-root> pwd
/sys/block/sdb/queue
Wed Mar 09 17:58:31

BTW, I looked into tagged command queuing for this system as well,
but apparently VIA SATA doesn't support TCQ on linux 2.6.x.
--------------------------------------------------------------------------------
Eventually the reblock | mtee egrep egrep gave:
egrep: memory exhausted
...using GNU egrep 2.5.1.
...so now I'm trying something closer to my classical method:
seki-root> reblock -e 75216016 $(expr 1024 \* 1024) 300 <
/dev/mapper/VolGroup00-LogVol00 | mtee './bgrep dptutil | ./ranges >
dptutil-ranges' './bgrep dptmgr | ./ranges > dptmgr-ranges'
Creating 2 pipes
popening ./bgrep dptutil | ./ranges > dptutil-ranges
popening ./bgrep dptmgr | ./ranges > dptmgr-ranges
stdin seems seekable, but file length is 0 - no exact percentages
Estimated filetransfer size is 77021200384 bytes
Estimated percentages will only be as accurate as your size estimate
(estimate: 1.3%  16s 12m 1h) Kbytes: 1027072.0  Mbits/s: 133.6  Gbytes/hr:
58.7  min: 1.0
(estimate: 2.5%  36s 16m 1h) Kbytes: 1913856.0  Mbits/s: 124.5  Gbytes/hr:
54.7  min: 2.0
(estimate: 3.7%  10s 17m 1h) Kbytes: 2814976.0  Mbits/s: 122.1  Gbytes/hr:
53.6  min: 3.0
(estimate: 4.9%  10s 17m 1h) Kbytes: 3706880.0  Mbits/s: 120.6  Gbytes/hr:
53.0  min: 4.0
...
--------------------------------------------------------------------------------
I've added a -s option to reblock, which makes it sleep for an  arbitrary
number of (fractions of) seconds between blocks.  Between this and the
I/O scheduler change, seki has become very pleasant to work on again,
despite the hunt for my missing palm memo.  :)
--------------------------------------------------------------------------------
From Bryan Ragon 

Here is a detailed list of steps that worked:

;; first backed up the first 512 bytes of /dev/hdb
# dd if=/dev/hdb of=~/hdb.first512 count=1 bs=512
1+0 records in
1+0 records out
 

;; zero them out, per Alasdair
# dd if=/dev/zero of=/dev/hdb count=1 bs=512
1+0 records in
1+0 records out

;; verified
# blockdev --rereadpt /dev/hdb
BLKRRPART: Input/output error

;; find the volumes
# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "media_vg" using metadata type lvm2

# pvscan
  PV /dev/hdb   VG media_vg   lvm2 [111.79 GB / 0    free]
  Total: 1 [111.79 GB] / in use: 1 [111.79 GB] / in no VG: 0 [0   ]

# lvmdiskscan
  /dev/hda1 [      494.16 MB]
  /dev/hda2 [        1.92 GB]
  /dev/hda3 [       18.65 GB]
  /dev/hdb  [      111.79 GB] LVM physical volume
  /dev/hdd1 [       71.59 GB]
  0 disks
  4 partitions
  1 LVM physical volume whole disk
  0 LVM physical volumes

# vgchange -a y
  1 logical volume(s) in volume group "media_vg" now active

;; /media is a defined mount point in fstab, listed below for future archive
searches
# mount /media
# ls /media
graphics  lost+found  movies  music


Success!!  Thank you, Alasdair!!!!

/etc/fstab

/dev/media_vg/media_lv  /media          ext3            noatime
0 0

--------------------------------------------------------------------------------
home blee has:
hdc1 ext3 /big wdc
sda5 xfs /backups
00/00 ext3 hda ibm fc3: too hot?
00/01 swap hda ibm
01/00 ext3 hdd maxtor fc4
01/01 swap hdd maxtor
hdb that samsung dvd drive that overheats
--------------------------------------------------------------------------------
 

[Aug 26, 2008] Moving LVM volumes to a different volume group - Lone Wolves - Web, game, and open source development by Sander Marechal

2008-08-25 | www.jejik.com

I went with SystemRescueCD which comes with both mdadm and LVM out-of-the-box.

The system layout is quite simple. /dev/sda1 and /dev/sdb1 make up a 500 GB mdadm RAID1 volume. This RAID volume contains an LVM volume group called “3ware”, named so because in my old server it was connected to my 3ware RAID card. It contains a single logical volume called “media”. The original 80 GB disk is on /dev/sdc1 which contains an LVM volume group called “linuxvg”. Inside that volume group are three volumes: “boot”, “root” and “swap”. Goal: Move linuxvg-root and linuxvg-boot to the 3ware volume group. Additional goal: Rename 3ware to linuxvg. The latter is more for aesthetic reasons but as a bonus it also means that there is no need to fiddle with grub or fstab settings after the move.

Before starting SystemRescueCD and start moving things around there are a few things that need to be done first. Start by making a copy of /etc/mdadm/mdadm.conf because you will need it later. Also, because the machine will be booting from the RAID array I need to install grub to those two disks.

# grub-install /dev/sda
# grub-install /dev/sdb

Now it’s time to boot into SystemRescueCD. I start off by copying /etc/mdadm/mdadm.conf back and starting the RAID1 array. This command scans for all the arrays defined in mdadm.conf and tries to start them.

# mdadm --assemble --scan

Next I need to make a couple of changes to /etc/lvm/lvm.conf. If I were to scan for LVM volume groups at this point, it would find the 3ware group three times: once in /dev/md0, /dev/sda1 and /dev/sdb1. So I adjust the filter setting in lvm.conf so it will not scan /dev/sda1 and /dev/sdb1.

filter = [ "r|/dev/cdrom|", "r|/dev/sd[ab]1|" ]

LVM can now scan the hard drives and find all the volume groups.

# vgscan

I disable the volume groups so that I can rename them. linuxvg becomes linuxold and 3ware becomes the new linuxvg. Then I re-enable the volume groups.

# vgchange -a n
# vgrename linuxvg linuxold
# vgrename 3ware linuxvg
# vgchange -a y

Now I can create a new logical volume in the 500 Gb volume group for my boot partition and create an ext3 filesystem in it.

# lvcreate --name boot --size 512MB linuxvg
# mkfs.ext3 /dev/mapper/linuxvg-boot

I create mount points to mount the original boot partition and the new boot partition and then use rsync to copy all the data. Don’t use cp for this! Rsync with the -ah option will preserve all soft links, hard links and file permissions while cp does not. If you do not want to use rsync you could also use the dd command to transfer the data directly from block device to block device.

# mkdir /mnt/src /mnt/dst
# mount -t ext3 /dev/mapper/linuxold-boot /mnt/src
# mount -t ext3 /dev/mapper/linuxvg-boot /mnt/dst
# rsync -avh /mnt/src/ /mnt/dst/
# umount /mnt/src /mnt/dst

Rinse and repeat to copy over the root filesystem.

# lvcreate --name root --size 40960MB linuxvg
# mkfs.ext3 /dev/mapper/linuxvg-root
# mount -t ext3 /dev/mapper/linuxold-root /mnt/src
# mount -t ext3 /dev/mapper/linuxvg-root /mnt/dst
# rsync -avh /mnt/src/ /mnt/dst/
# umount /mnt/src /mnt/dst

There's no sense in copying the swap volume. Simply create a new one.

# lvcreate --name swap --size 1024MB linuxvg
# mkswap /dev/mapper/linuxvg-swap

And that's it. I rebooted into Debian Lenny to make sure that everything worked and I removed the 80 GB disk from my server. While this wans’t particularly hard, I do hope that the maintainers of LVM create an lvmove command to make this even easier.

[Aug  15, 2008] Linux RAID Smackdown Crush RAID 5 with RAID 10

LinuxPlanet

Creating RAID 10

No Linux installer that I know of supports RAID 10, so we have to jump through some extra hoops to set it up in a fresh installation. This is my favorite layout for RAID systems:

• /dev/md0 is a RAID 1 array containing the root filesystem.
• /dev/md1 is a RAID 10 array containing a single LVM group divided into logical volumes for /home, /var, and /tmp, and anything else I feel like stuffing in there.
• Each disk has its own identical swap partition that is not part of RAID or LVM, just plain old ordinary swap.

One way is to use your Linux installer to create the RAID 1 array and the swap partitions, then boot into the new filesystem and create the RAID 10 array. This works, but then you have to move /home, /var, /tmp, and whatever you else you want there, which means copying files and editing /etc/fstab. I get tired thinking about it.

Another way is to prepare your arrays and logical volumes in advance and then install your new system over them, and that is what we are going to do. You need a bootable live Linux that includes mdadm, LVM2 and GParted, unless you're a crusty old command-line commando that doesn't need any sissy GUIs, and are happy with fdisk. Two that I know have all of these are Knoppix and SystemRescueCD; I used SystemRescueCD.

Step one is to partition all of your drives identically. The partition sizes in my example system are small for faster testing; on a production system the 2nd primary partition would be as large as possible:

• 1st primary partition, 5GB
• 2nd primary partition, 7GB
• swap partition, 1GB

The first partition on each drive must be marked as bootable, and the first two partitions must be marked as "fd Linux raid auto" in fdisk. In GParted, use Partition -> Manage Flags.

Now you can create your RAID arrays with the mdadm command. This command creates the RAID1 array for the root filesystem:

# mdadm -v --create /dev/md0 --level=raid1 --raid-devices=2 /dev/hda1 /dev/sda1
mdadm: layout defaults to n1
mdadm: chunk size defaults to 64K
mdadm: size set to 3076352K
mdadm: array /dev/md0 started.

This will take some time, which cat /proc/mdstat will tell you:

Personalities : 'linear' 'raid0' 'raid1' 'raid6' 'raid5' 'raid4' 'multipath' 'raid10' md0 : active raid10 sda1'1' hda1'0'
3076352 blocks 2 near-copies '2/2' 'UU'
'====>................' resync = 21.8% (673152/3076352) finish=3.2min speed=12471K/sec

This command creates the RAID 10 array:

# mdadm -v --create /dev/md1 --level=raid10 --raid-devices=2 /dev/hda2 /dev/sda2

Naturally you want to be very careful with your drive names, and give mdadm time to finish. It will tell you when it's done:

RAID10 conf printout:
--- wd: rd:2
disk 0, wo:0, o:1, dev:hda2
disk 1, wo:0, o:1, dev:sda2

mdadm --detail /dev/md0 displays detailed information on your arrays.

Create LVM Group and Volumes

Now we'll put a LVM group and volumes on /dev/md1. I use vg- for volume group names and lv- for the logical volumes in the volume groups. Using descriptive names, like lv-home, will save your sanity later when you're creating filesystems and mountpoints. The -L option specifies the size of the volume:

# pvcreate /dev/md1
# vgcreate vg-server1 /dev/md1
# lvcreate -L4g -nlv-home vg-server1
# lvcreate -L2g -nlv-var vg-server1
# lvcreate -L1g -nlv-tmp vg-server1

 

You'll get confirmations for every command, and you can use vgdisplay and lvdisplay to see the fruits of your labors. Use vgdisplay to see how much space is left.

[Jul 26, 2008] Recovery of RAID and LVM2 Volumes by Richard Bullington-McGuire

April 28th, 2006 | Linux Journal

The combination of Linux software RAID (Redundant Array of Inexpensive Disks) and LVM2 (Logical Volume Manager, version 2) offered in modern Linux operating systems offers both robustness and flexibility, but at the cost of complexity should you ever need to recover data from a drive formatted with software RAID and LVM2 partitions. I found this out the hard way when I recently tried to mount a system disk created with RAID and LVM2 on a different computer. The first attempts to read the filesystems on the disk failed in a frustrating manner.

I had attempted to put two hard disks into a small-form-factor computer that was really only designed to hold only one hard disk, running the disks as a mirrored RAID 1 volume. (I refer to that system as raidbox for the remainder of this article.) This attempt did not work, alas. After running for a few hours, it would power-off with an automatic thermal shutdown failure. I already had taken the system apart and started re-installing with only one disk when I realized there were some files on the old RAID volume that I wanted to retrieve.

Recovering the data would have been easy if the system did not use RAID or LVM2. The steps would have been to connect the old drive to another computer, mount the filesystem and copy the files from the failed volume. I first attempted to do so, using a computer I refer to as recoverybox, but this attempt met with frustration.

On July 18th, 2007 Richard Bullington-McGuire says:

> ... couldn't you have booted the recovery box with a Live CD and simply mounted

only the drive partitions you needed?

That was what I was originally hoping to do, but that did not work automatically. RAID arrays on USB-connected drives are not available to the system when it does its first scan for RAID arrays. Also, if the recovery box has a volume group with the same name, it will not recognize the newly-attached volume group.

I have used USB RAID arrays in production, and you have to take some extra steps to activate them late in the boot process. I typically use a script similar to this to do the job:


#!/bin/sh
#
# Mount a USB raid array
#
# Call from /etc/rc.d/rc.local

# Activate the array. This assumes that /etc/mdadm.conf has an entry for it already
/sbin/mdadm -A -s
# Look for LVM2 volume groups on all connected partitions, including the array
/sbin/vgscan --mknodes
# Activate all LVM partitions, including that on the array
/sbin/vgchange -a y
# Make sure to fsck the device so it stays healthy long-term
fsck -T -a $DEVICE
mount $DEVICE $MOUNTPOINT
 

> In otherwords, just don't mount the drive in the recovery box that had the equivalent vol group. That way there would have been no conflict right?

That's mostly right. You'd still need to scan for the RAID arrays with 'mdadm --examine --scan $MYDEVICENAME' , then activate them after creating /etc/mdadm.conf.

If you had other md software RAID devices on the system, you might have to fix up the device numbering on the md devices.

> If the recovery box did NOT have any LVM partitions or LVM config native to it.. could i simply plug the raid drive in and the recovery box would automagically find the raid LVM partitions or would I still have to something else to make it work?

On a recovery box without any software RAID or LVM configuration, if you plugged the RAID drive directly into the IDE or SATA connector, it might automagically find the RAID array and LVM volume. I have not done that particular experiment, you might try it and let me know how it goes.

If the drive was attached to the recovery box using a USB enclosure, the RAID and LVM configurations probably won't be autodetected during the early boot stages, and you'll almost certainly have to do a scan / activate procedure on both the RAID and LVM layers.

You might have to scan for RAID partitions, build an /etc/mdadm.conf file, and then scan for volume groups and activate them in either case.

The most difficult part of the recovery outlined in the article was pulling the LVM configuration out of the on-disk ring buffer. You can avoid that by making sure you have a backup of the LVM configuration for that machine stored elsewhere.

• reply
• Printer-friendly version

Experienced this exact

On September 13th, 2006 Neekofab (not verified) says:

bleah

• reply
• Printer-friendly version

I ran into the same issue

On May 9th, 2007 Anonymous (not verified) says:

# MAKEDEV md1
# mdadm -C /dev/md1 -l 1 -n 2 missing /dev/sdb1

Voila. Your raid array has now been moved from md0 to md1.

Help restoring LVM partition Redhat, RHEL 5, 5.1, LVM

Example of wrong move then destroys LVM control table. You need to use vgextend VolGroup00 /dev/sdb1 to add new disk to existing logical volume.  See Recovery of RAID and LVM2 Volumes for information about recovery.

Zones: Linux, Linux Administration, Linux Setup

Tags: Redhat, RHEL 5, 5.1, LVM

 

I was working on a Dell Precision Workstation system with 2 SAS drives.
The first drive /dev/sda was partitioned with the following table

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14       36472   292856917+  8e  Linux LVM

[root@lrc200604665 tsm]# cat /etc/fstab
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

I wanted to add a second hard drive to the system and mount it as "home"

My idea was to add it to the existing Volume Group VolGroup00

After I formated the drive and using the standard Linux 8E LVM2 partition type,  I ran the following  command to prepare it for LVM

[root@lrc200604665 home]# pvcreate  /dev/sdb1
  Can't initialize physical volume "/dev/sdb1" of volume group "VolGroup00" without -ff

Well,  I ran the command and it overwrote my LVM table.

pvscan detects a UUID mitchmatch since  PVCREATE overwrote the VolGroup00
vgscan and lvscan are also useless.

The system will  odiously not boot now

any help would be greatly appreciated

Red Hat Magazine Tips and tricks What is the procedure to resize an LVM2 logical volume and the ext2 or ext3 filesystem

It is recommended that a resizing procedure be tested before performing it on a filesystem that contains critical data. It is also strongly recommended that data backups are created and verified before resizing filesystems. To assist the reader in becoming familiar with the resizing procedure, the rest of this article describes a scenario in which un-partitioned disk space is used to test logical volume and ext3 filesystem resizing.

In this test procedure, first we create a new partition (/dev/hda2) using un-partitioned disk space. A new volume group (TestVG) and logical volume (TestLV) are then created using a physical volume created on the partition. An ext3 filesystem is then created on the logical volume, it is checked, and then mounted. A new physical volume (/dev/hda3) using the rest of the un-partioned disk space is then created and added to the volume group and logical volume. Finally, the filesystem on the logical volume is resized, and the integrity of the filesystem is checked.

In this test procedure, the parted, lvm, mount, mkfs.ext3, df, e2fsck and ext2online programs are used. The reader is encouraged to become familiar with these programs by consulting the manual pages.

Getting E-mail notifications when MD devices fail

I use the MD (multiple device) logical volume manager to mirror the boot devices on the Linux servers I support. When I first started using MD, the mdadm utility was not available to manage and monitor MD devices. Since disk failures are relatively common in large shops, I used the shell script from my SysAdmin article Monitoring and Managing Linux Software RAID to send E-mail when a device entered the failed state. While reading through the mdadm(8) manual page, I came across the “–monitor” and “–mail” options. These options can be used to monitor the operational state of the MD devices in a server, and generate E-mail notifications if a problem is detected. E-mail notification support can be enabled by running mdadm with the “–monitor” option to monitor devices, the “–daemonise” option to create a daemon process, and the “–mail” option to generate E-mail:

$ /sbin/mdadm –monitor –scan –daemonise –mail=root@localhost

Once mdadm is daemonized, an E-mail similar to the following will be sent each time a failure is detected:

From: mdadm monitoring 
To: root@localhost.localdomain
Subject: Fail event on /dev/md1:biscuit

This is an automatically generated mail message from mdadm
running on biscuit

A Fail event had been detected on md device /dev/md1.

Faithfully yours, etc.

I digs me some mdadm!

Linux LVM silliness

While attempting to create a 2-way LVM mirror this weekend on my Fedora Core 5 workstation, I received the following error:

$ lvcreate -L1024 -m 1 vgdata

  Not enough PVs with free space available for parallel allocation.
  Consider --alloc anywhere if desperate.

Since the two devices were initialized specifically for this purpose and contained no other data, I was confused by this error message. After scouring Google for answers, I found a post that indicated that I needed a log LV for this to work, and the log LV had to be on it’s own disk. I am not sure about most people, but who on earth orders a box with three disks? Ugh!

Posted by matty, filed under Linux LVM. Date: May 3, 2006, 9:50 pm | 2 Comments

[linux-lvm] Raid 0+1

[linux-lvm] Raid 0+1

• From: "Wayne Pascoe" <lists-june2004 penguinpowered org>
• To: linux-lvm redhat com
• Subject: [linux-lvm] Raid 0+1
• Date: Wed, 21 Jul 2004 13:22:53 +0100 (BST)

Hi all,

I am working on a project to evaluate LVM2 against Veritas Volume
Manager for a new Linux deployment. I am trying to get a Raid 0+1
solution working and I'm struggling.

So far, this is where I am:

1. I created 8GB partitions on 4 disks, sdb, sdc, sdd and sde, and set
their partition types to 8e with fdisk

2. I then ran vgscan, follwed by pvcreate /dev/sdb1, /dev/sdc1,
/dev/sdd1, /dev/sde1

3. Next, I created 2 volume groups as follows:
vgcreate StripedData1 /dev/sdb1 /dev/sdc1
vgcreate StripedData2 /dev/sdd1 /dev/sde1

4. Next, I created 2 volumes, one in each group as follows:
lvcreate -i 2 -I 64 -n Data1 -L 6G StripedData1
lvcreate -i 2 -I 64 -n Data2 -L 6G StripedData2
Now I have 2 striped volumes, but no redundancy. This is where I think
things start to go wrong.

5. I now create a raid device, /dev/md0 consisting of these two
volumes. I run mkraid on this, create a file system, and mount it on
/Data1. This all works fine, and I have a 6GB filesystem on /Data1

Now I need to be able to resize this whole solution, and I'm not sure
if the way I've built it caters for what I need to do...

I unmount /Data1 and use lvextend to extend the 2 volumes from 6GB to
7.5GB. This succeeds. Now even though both of the volumes that make up
/dev/md0 are extended, I cannot resize /dev/md0 using resize2fs
/dev/md0

Can anyone advise me how I can achieve what I'm looking for here ? I'm
guessing maybe I did things the wrong way around, but I can't find a
solution that will give me both striping and mirroring :(

Thanks in advance,


-- 
Wayne Pascoe

LVM HOWTO

Introduction

1. Latest Version

2. Disclaimer

3. Contributors

1. What is LVM?

2. What is Logical Volume Management?

2.1. Why would I want it?

2.2. Benefits of Logical Volume Management on a Small System

2.3. Benefits of Logical Volume Management on a Large System

3. Anatomy of LVM

3.1. volume group (VG)

3.2. physical volume (PV)

3.3. logical volume (LV)

3.4. physical extent (PE)</ mapping modes (linear/striped)

3.8. Snapshots

4. Frequently Asked Questions

4.1. LVM 2 FAQ

4.2. LVM 1 FAQ

5. Acquiring LVM

5.1. Download the source

5.2. Download the development source via CVS

5.3. Before You Begin

5.4. Initial Setup

5.5. Checking Out Source Code

5.6. Code Updates

5.7. Starting a Project

5.8. Hacking the Code

5.9. Conflicts

6. Building the kernel modules

6.1. Building the device-mapper module

6.2. Build the LVM 1 kernel module

7. LVM 1 Boot time scripts

7.1. Caldera

7.2. Debian

7.3. Mandrake

7.4. Redhat

7.5. Slackware

7.6. SuSE

8. LVM 2 Boot Time Scripts

9. Building LVM from the Source

9.1. Make LVM library and tools

9.2. Install LVM library and tools

9.3. Removing LVM library and tools

10. Transitioning from previous versions of LVM to LVM 1.0.8

10.1. Upgrading to LVM 1.0.8 with a non-LVM root partition

10.2. Upgrading to LVM 1.0.8 with an LVM root partition and initrd

11. Common Tasks

11.1. Initializing disks or disk partitions

11.2. Creating a volume group

11.3. Activating a volume group

11.4. Removing a volume group

11.5. Adding physical volumes to a volume group

11.6. Removing physical volumes from a volume group

11.7. Creating a logical volume

11.8. Removing a logical volume

11.9. Extending a logical volume

11.10. Reducing a logical volume

11.11. Migrating data off of a physical volume

12. Disk partitioning

12.1. Multiple partitions on the same disk

12.2. Sun disk labels

13. Recipes

13.1. Setting up LVM on three SCSI disks

13.2. Setting up LVM on three SCSI disks with striping

13.3. Add a new disk to a multi-disk SCSI system

13.4. Taking a Backup Using Snapshots

13.5. Removing an Old Disk

13.6. Moving a volume group to another system

13.7. Splitting a volume group

13.8. Converting a root filesystem to LVM 1

13.9. Recover physical volume metadata

A. Dangerous Operations

A.1. Restoring the VG UUIDs using uuid_fixer

A.2. Sharing LVM volumes

B. Reporting Errors and Bugs

C. Contact and Links

C.1. Mail lists

C.2. Links

D. GNU Free Documentation License

D.1. PREAMBLE

D.2. APPLICABILITY AND DEFINITIONS

D.3. VERBATIM COPYING

D.4. COPYING IN QUANTITY

D.5. MODIFICATIONS

D.6. COMBINING DOCUMENTS

D.7. COLLECTIONS OF DOCUMENTS

D.8. AGGREGATION WITH INDEPENDENT WORKS

D.9. TRANSLATION

D.10. TERMINATION

D.11. FUTURE REVISIONS OF THIS LICENSE

D.12. ADDENDUM: How to use this License for your documents

The Linux and Unix Menagerie LVM Quick Command Reference For Linux And Unix

1. LVM Basic relationships. A quick run-down on how the different parts are related

Physical volume - This consists of one, or many, partitions (or physical extent groups) on a physical drive.
Volume group - This is composed of one or more physical volumes and contains one or more logical volumes.
Logical volume - This is contained within a volume group.

2. LVM creation commands (These commands are used to initialize, or create, new logical objects) - Note that we have yet to explore these fully, as they can be used to do much more than we've demonstrated so far in our simple setup.

pvcreate - Used to create physical volumes.
vgcreate - Used to create volume groups.
lvcreate - Used to create logical volumes.

3. LVM monitoring and display commands (These commands are used to discover, and display the properties of, existing logical objects). Note that some of these commands include cross-referenced information. For instance, pvdisplay includes information about volume groups associated with the physical volume.

pvscan - Used to scan the OS for physical volumes.
vgscan - Used to scan the OS for volume groups.
lvscan - Used to scan the OS for logical volumes.
pvdisplay - Used to display information about physical volumes.
vgdisplay - Used to display information about volume groups.
lvdisplay - Used to display information about logical volumes.

4. LVM destruction or removal commands (These commands are used to ensure that logical objects are not allocable anymore and/or remove them entirely) Note, again, that we haven't fully explored the possibilities with these commands either. The "change" commands in particular are good for a lot more than just prepping a logical object for destruction.

pvchange - Used to change the status of a physical volume.
vgchange - Used to change the status of a volume group.
lvchange - Used to change the status of a logical volume.
pvremove - Used to wipe the disk label of a physical drive so that LVM does not recognize it as a physical volume.
vgremove - Used to remove a volume group.
lvremove - Used to remove a logical volume.

5. Manipulation commands (These commands allow you to play around with your existing logical objects. We haven't posted on "any" of these commands yet - Some of them can be extremely dangerous to goof with for no reason)

pvextend - Used to add physical devices (or partition(s) of same) to a physical volume.
pvreduce - Used to remove physical devices (or partition(s) of same) from a physical volume.
vgextend - Used to add new physical disk (or partition(s) of same) to a volume group.
vgreduce - Used to remove physical disk (or partition(s) of same) from a volume group.
lvextend - Used to increase the size of a logical volume.
lvreduce - Used to decrease the size of a logical volume.
 

The Linux and Unix Menagerie Monitoring and Display Commands For LVM On Linux And Unix

Physical Volumes:

The two commands we'll be using here are pvscan and pvdisplay.

pvscan, as with all of the following commands, pretty much does what the name implies. It scans your system for LVM physical volumes. When used straight-up, it will list out all the physical volumes it can find on the system, including those "not" associated with volume groups (output truncated to save on space):

host # pvscan
pvscan
pvscan -- reading all physical volumes (this may take a while...)
...
pvscan -- ACTIVE PV "/dev/hda1" is in no VG [512 MB]
...
pvscan -- ACTIVE PV "/dev/hdd1" of VG "vg01"[512 MB / 266 MB free]
...

Next, we'll use pvdisplay to display our only physical volume:

host # pvdisplay /dev/hdd1 <-- Note that you can leave the /dev/hdd1, or any specification, off of the command line if you want to display all of your physical volumes. We just happen to know we only have one and are being particular ;)

...
PV Name /dev/hdd1
VG Name vg01
PV Size 512 MB
...

Other output should include whether or not the physical volume is allocatable (or "can be used" ;), total physical extents (see our post on getting started with LVM for a little more information on PE's), free physical extents, allocated physical extents and the physical volume's UUID (Identifier).

Volume Groups:

The two commands we'll be using here are vgscan and vgdisplay.

vgscan will report on all existing volume groups, as well as create a file (generally) called /etc/lvmtab (Some versions will create an /etc/lvmtab.d directory as well):

 

host # vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found active volume group "vg01"
...

vgdisplay can be used to check on the state and condition of our volume group(s). Again, we're specifying our volume group on the command line, but this is not necessary:

 

host # vgdisplay vg01
...
VG Name vg01
...
VG Size 246 MB
...

this command gives even more effusive output. Everything from the maximum logical volumes the volume group can contain (including how many it currently does and how many of those are open), separate (yet similar) information with regards to the physical volumes it can encompass, all of the information you've come to expect about the physical extents and, of course, each volume's UUID.
 

[Aug 11, 2007] Logical volume management by Klaus Heinrich Kiwi

11 Sep 2007 | IBM developerworks

Volume management is not new in the -ix world (UNIX®, AIX, and so forth). And logical volume management (LVM) has been around since Linux® kernel 2.4v1 and 2.6.9v2. This article reveals the most useful features of LVM2—a relatively new userspace toolset that provides logical volume management facilities—and suggests several ways to simplify your system administration tasks.

Logical volume management (LVM) is a way systems can abstract physical volume management into a higher-level and usually simpler paradigm. By using LVM, all physical disks and partitions, no matter what size and how scattered they are, can be abstracted and viewed as a single storage source. For example, in the layout of physical-to-logical mapping shown in Figure 1, how could the user create a filesystem of, say 150GB, since the biggest disk is 80GB large?

 

By aggregating partitions and whole disks into a virtual disk, LVM can sum small storage spaces into a bigger, consolidated one. This virtual disk, in LVM terms, is called volume group.

And the possibility of having a filesystem bigger than your biggest disk isn't the only magic feature of this high-level paradigm of storage management. With LVM, you can also:

• Add disks/partitions to your disk-pool and extend existing filesystems online
• Replace two 80GB disks with one 160GB disk without the need to bring the system offline or manually move data between disks
• Shrink filesystems and remove disks from the pool when their storage space is no longer necessary
• Perform consistent backups using snapshots (more on this later in the article)

LVM2 refers to a new userspace toolset that provides logical volume management facilities In Linux. It is fully backwards-compatible with the original LVM toolset. In this article, you'll see the most useful features of LVM2 as well as some other possible uses to simplify your system administration tasks. (By the way, if you're looking for a more basic guide to LVM, try the LVM HowTo listed in the Resources section below).

Let's look at how the LVM is organized.

LVM organization

The LVM is structured in three elements:

• Volumes: physical and logical volumes and volume groups
• Extents: physical and logical extents
• Device mapper: the Linux kernel module

Volumes

Linux LVM is organized into physical volumes (PVs), volume groups (VGs), and logical volumes (LVs). Physical volumes are physical disks or physical disk partitions (as in /dev/hda or /dev/hdb1). A volume group is an aggregation of physical volumes. And a volume group can be logically partitioned into logical volumes.

 

All four partitions in physical disk 0 (/dev/hda[1-4]), as well as the whole of physical disk 1 (/dev/hdb) and physical disk 2 (/dev/hdd), were added as PVs to volume group VG0.

The volume group is where the magic of n-to-m mapping is done (as in, n PVs can be seen as m LVs). So after the assignment of PVs to the volume group, you can create a logical volume of any size (to the maximum of the VG size). In the example in Figure 2, a volume group named LV0 was created, leaving some free-space for other LVs (or for posterior LV0 growth).

Logical volumes are the LVM equivalent of physical disks partitions—for all practical purposes, they are physical disk partitions.

So, after the creation of an LV, you can use it with whatever filesystem you prefer and mount it under some mount point to start using it. Figure 3 shows a formatted logical volume, LV0, mounted under /var.

 

Extents

In order to do the n-to-m, physical-to-logical volumes mapping, PVs and VGs must share a common quantum size for their basic blocks; these are called physical extents (PEs) and logical extents (LEs). Despite the n-physical to m-logical volume mapping, PEs and LEs always map 1-to-1.

With LVM2, there's no limit on the maximum numbers of extents per PV/LV. The default extent size is 4MB, and there's no need to change this for most configurations, because there is no I/O performance penalty for smaller/bigger extent size. LVM tools usage, however, can suffer from high extent count, so using bigger extents can keep the extent count low. Be aware, however, that different extent sizes can't be mixed in a single VG, and changing the extent size is the single unsafe operation with the LVM: It can destroy data. The best advice is to stick with the extent size chosen in the initial setup.

Different extent sizes means different VG granularity. For instance, if you choose an extent size of 4GB, you can only shrink/extend LVs in steps of 4GB.

Figure 4 shows the same layout used in previous examples with the PEs and LEs shown (the free space inside VG0 is also formed of free LEs, even though they're not shown).

 

Also note the extent allocation policy in Figure 4. LVM2 doesn't always allocate PEs contiguously; for more details, see the Linux man page on lvm (see the Resources below for a link). The system administrator can set different allocation policies, but that isn't normally necessary, since the default one (called the normal allocation policy) uses common-sense rules such as not placing parallel stripes on the same physical volume.

If you decide to create a second LV (LV1), the final PE distribution may look like the one shown in Figure 5.

 

Device mapper

Device mapper (also known as dm_mod) is a Linux kernel module (it can be built-in too), upstream since kernel 2.6.9. Its job (as the name says) is to map devices—it is required by LVM2.

In most major distributions, Device mapper comes installed by default, and it is usually loaded automatically at boot time or when LVM2/EVMS packages are installed or enabled (EVMS is an alternative tool; more on that in Resources). If not, try to modprobe for dm_mod and then check your distro's documentation for how to enable it at boot time: modprobe dm_mod.

When creating VGs and LVs, you can give them a meaningful name (as opposed to the previous examples where, for didactic purposes, the names VG0, LV0, and LV1 were used). It is the Device mapper's job to map these names correctly to the physical devices. Using the previous examples, the Device mapper would create the following device nodes in the /dev filesystem:

• /dev/mapper/VG0-LV0
  • /dev/VG0/LV0 is a link to the above
• /dev/mapper/VG0-LV1
  • /dev/VG0/LV1 is a link to the above

(Notice the name format standard: /dev/{vg_name}/{lv_name} -> /dev/mapper/{vg_name}{lv_name}).

As opposed to a physical disk, there's no raw access to a volume group (meaning there's no such thing as a /dev/mapper/VG0 file or you can't dd if=/dev/VG0 of=dev/VG1). You usually deal with these using the lvm(8) command(s).

Common tasks

Some common tasks you'll perform with LVM2 are systems verification (is LVM2 installed?) and volume creation, extension, and management.

Is your system ready for LVM2?

Verify whether your distro LVM2 package is installed. If not, install it (always giving preference to your original packages).

The Device mapper module must be loaded at system startup. Check to see if it is currently loaded with lsmod | grep dm_mod. Otherwise, you might need to install and configure additional packages (the original documentation can show you how to enable LVM2).

If you're just testing things (or maybe rescuing a system), use these basic commands to start using LVM2:

Listing 1. Basic commands to fire up LVM2
 

#this should load the Device-mapper module modprobe dm_mod #this should find all the PVs in your physical disks pvscan #this should activete all the Volume Groups vgchange -ay

 

If you plan to have your root filesystem inside an LVM LV, take extra care with the initial-ramdisk image. Again, the distros usually take care of this—when installing the LVM2 package, they usually rebuild or update the initrd image with the appropriate kernel modules and activation scripts. But you may want to browse through your distro documentation and make sure that LVM2 root filesystems are supported.

Note that the initial-ramdisk image usually activates LVM only when it detects that the root filesystem is under a VG. That's usually done by parsing the root= kernel parameter. Different distros have different ways to determine whether the root filesystem path is or is not inside a volume group. Consult your distro documentation for details. If unsure, check your initrd or initramdisk configuration.

Create new volumes

Using your favorite partitioner (fdisk, parted, gparted), create a new partition for LVM usage. Although supported by LVM, using an LVM on top of an entire disk is not recommended: Other operating systems may see this disk as uninitialized and wipe it out! Better to create a partition covering the whole disk.

Most partitioners usually default to create new partitions using the 0x83 (or Linux) partition ID. You can use the default, but for organization purposes, it is better to change it to 0x8e (or Linux LVM).

After you've created a partition, you should see one (or more) Linux LVM partitions in your partition table:

root@klausk:/tmp/a# fdisk -l Disk /dev/hda: 80.0 GB, 80026361856 bytes 255 heads, 63 sectors/track, 9729 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 1623 13036716 7 HPFS/NTFS /dev/hda2 1624 2103 3855600 8e Linux LVM /dev/hda3 2104 2740 5116702+ 83 Linux /dev/hda4 3000 9729 54058725 5 Extended /dev/hda5 9569 9729 1293232+ 82 Linux swap / Solaris /dev/hda6 3000 4274 10241374+ 83 Linux /dev/hda7 4275 5549 10241406 83 Linux /dev/hda8 5550 6824 10241406 83 Linux /dev/hda9 6825 8099 10241406 83 Linux /dev/hda10 8100 9568 11799711 8e Linux LVM Partition table entries are not in disk order root@klausk:/tmp/a#

 

Now initialize each partition with pvcreate:

Listing 2. Initializing partitions
 

root@klausk:/tmp/a# pvcreate /dev/hda2 /dev/hda10 Physical volume "/dev/hda2" successfully created Physical volume "/dev/hda10" successfully created root@klausk:/tmp/a#

 

The PVs and the VG are created in a single step: vgcreate:

Listing 3. Making PVs and the VG
 

root@klausk:~# vgcreate test-volume /dev/hda2 /dev/hda10 Volume group "test-volume" successfully created root@klausk:~#

 

The command above creates a logical volume called test-volume using /dev/hda2 and /dev/hda10 as the initial PVs.

After the VG test-volume creation, use the vgdisplay command to review general info about the newly created VG:

Listing 4. Checking out general info on your new VG
 

root@klausk:/dev# vgdisplay -v test-volume Using volume group(s) on command line Finding volume group "test-volume" --- Volume group --- VG Name test-volume System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 2 Act PV 2 VG Size 14.93 GB PE Size 4.00 MB Total PE 3821 Alloc PE / Size 0 / 0 Free PE / Size 3821 / 14.93 GB VG UUID lk8oco-ndQA-yIMZ-ZWhu-LtYX-T2D7-7sGKaV --- Physical volumes --- PV Name /dev/hda2 PV UUID 8LTWlw-p1OJ-dF6w-ZfMI-PCuo-8CiU-CT4Oc6 PV Status allocatable Total PE / Free PE 941 / 941 PV Name /dev/hda10 PV UUID vC9Lwb-wvgU-UZnF-0YcE-KMBb-rCmU-x1G3hw PV Status allocatable Total PE / Free PE 2880 / 2880 root@klausk:/dev#

 

In Listing 4, check that there are 2 PVs assigned to this VG with the total size of 14.93GB (that is, 3,821 PEs of 4MB each)—don't forget to see that all of them are free for use!

Now that the volume group is ready to use, use it like a virtual disk to create/remove/resize partitions (LVs)—note that the Volume Group is an abstract entity, only seen by the LVM toolset. Create a new logical volume using lvcreate:

Listing 5. Making new logical volumes (partitions)
 

root@klausk:/# lvcreate -L 5G -n data test-volume Logical volume "data" created root@klausk:/#

 

Listing 5 creates a 5GB LV named data. After data has been created, you can check for its device node:

Listing 6. Checking LVs device node
 

root@klausk:/# ls -l /dev/mapper/test--volume-data brw-rw---- 1 root disk 253, 4 2006-11-28 17:48 /dev/mapper/test--volume-data root@klausk:/# ls -l /dev/test-volume/data lrwxrwxrwx 1 root root 29 2006-11-28 17:48 /dev/test-volume/data -> /dev/mapper/test--volume-data root@klausk:/#

 

You can also check for the LV properties with the lvdisplay command:

Listing 7. Discovering LV properties
 

root@klausk:~# lvdisplay /dev/test-volume/data --- Logical volume --- LV Name /dev/test-volume/data VG Name test-volume LV UUID FZK4le-RzHx-VfLz-tLjK-0xXH-mOML-lfucOH LV Write Access read/write LV Status available # open 0 LV Size 5.00 GB Current LE 1280 Segments 1 Allocation inherit Read ahead sectors 0 Block device 253:4 root@klausk:~#

 

As you probably noticed, the LV name/path for all practical purposes is /dev/{VG_name}/{LV_name}, as in /dev/test-volume/data. Besides being the target for the /dev/{VG_name}/{LV_name} link, don't use the /dev/mapper/{VG_name}-{LV_name} file. The majority of LVM commands are expecting something in the format /dev/{vg-name}/{lv-name} as the target specification for operation.

Finally, with the logical volume ready, format it with whatever filesystem you prefer and then mount it under the desired mount point:

Listing 8. Mounting the LV
 

root@klausk:~# mkfs.reiserfs /dev/test-volume/data root@klausk:~# mkdir /data root@klausk:~# mount -t reiserfs /dev/test-volume/data /data/ root@klausk:~# df -h /data Filesystem Size Used Avail Use% Mounted on /dev/mapper/test--volume-data 5.0G 33M 5.0G 1% /data root@klausk:~#

 

You may also want to edit your fstab(5) file to automatically mount the filesystem at boot time:

Listing 9. Automatic LV mount
 

#mount Logical Volume 'data' under /data /dev/test-volume/data /data reiserfs defaults 0 2

 

The logical volume is like a block device for all purposes, including but not limited to using it as a raw partition for databases. This is, in fact, a standard best practice if you want to perform consistent backups over a database using LVM snapshots.

Extend volumes

This is the easy part. If you have enough free space in the volume group, you just have to use lvextend in order to extend the volume. There's no need to unmount it. Afterwards, also extend the filesystem inside the logical volume (they are two separate things, remember). Depending on the filesystem you're using, it also can be extended online (that's it, while mounted!).

If you don't have enough space in your VG, you'll need to add additional physical disks first. To do that:

• Use a free partition to create a physical disk. It is recommended that you change the partition type to 0x8e (Linux LVM) for easy identification of LVM partitions/disks. Use pvcreate to initialize the Physical Disk: pvcreate /dev/hda3.
• Then, use vgextend to add it to an existing VG: vgextend test-volume /dev/hda2.

You can also create or add several physical disks at once with:

pvcreate /dev/hda2 /dev/hda3 /dev/hda5 vgextend test-volume /dev/hda2 /dev/hda3 /dev/hda5

 

Once you're ready with adding PVs and have sufficient space to grow your logical volume, use lvextend to extend the logical volume(s): lvextend -L 8G /dev/test-volume/data. This command extends the /dev/test-volume/data LV to the size of 8GB.

There are several useful parameters for lvextend:

• You can use -L +5G if you extend your LV in 5GB chunks (relative size).
• You can specify where you want this new extension to be placed (in terms of PVs); just append the PV you want to use to the command.
• You can also specify the absolute/relative extension size in terms of PEs.

Take a look at lvextend(8) for more details.

After extending the LV, don't forget to also extend the filesystem (so you can actually use the extra space). This can be done online (with the filesystem mounted), depending on the filesystem.

Listing 10 is an example of resizing an reiserfs(v3) with resize_reiserfs (which can be used on a mounted filesystem, by the way): resize_reiserfs /dev/test-volume/data.

Manage volumes

To manage volumes, you need to know how to reduce LVs and how to remove PVs.

Reducing logical volumes
You can reduce an LV in the same way you extend one, using the lvreduce command. From the LVM side, this operation can always be done with the volume online. One caveat: the majority of filesystems don't support online filesystem shrinking. Listing 10 demonstrates a sample procedure:

Listing 10. Reducing an LV
 

#unmount LV umount /path/to/mounted-volume #shrink filesystem to 4G resize_reiserfs -s 4G /dev/test-volume/data #reduce LV lvreduce -L 4G /dev/vg00/test

 

Be careful with sizes and units: the filesystem should not be longer than the LV!

Removing physical volumes
Imagine the following situation: You have a volume group with two 80GB disks, and you want to upgrade those to 160GB disks. With LVM, you can remove a PV from a VG in the same way they are added (that means online!). Notice, though, that you can't remove PVs that are being used in an LV. For those situations, there is a great utility called pvmove that can free PVs online so you can replace them easily. In a hot-swap environment, you can even swap all disks with no downtime at all!

pvmove's only requirement is a contiguous number of free extents in the VG equivalent to the number of extents to be moved out of a PV. There's no easy way to directly determine the largest free set of contiguous PEs, but you can use pvdisplay -m to display the PV allocation map:

Listing 11. Displaying the PV allocation map
 

#shows the allocation map pvdisplay -m --- Physical volume --- PV Name /dev/hda6 VG Name test-volume PV Size 4.91 GB / not usable 1.34 MB Allocatable yes (but full) PE Size (KByte) 4096 Total PE 1200 Free PE 0 Allocated PE 1200 PV UUID BA99ay-tOcn-Atmd-LTCZ-2KQr-b4Z0-CJ0FjO --- Physical Segments --- Physical extent 0 to 2367: Logical volume /dev/test-volume/data Logical extents 5692 to 8059 Physical extent 2368 to 2499: Logical volume /dev/test-volume/data Logical extents 5560 to 5691 --- Physical volume --- PV Name /dev/hda7 VG Name test-volume PV Size 9.77 GB / not usable 1.37 MB Allocatable yes PE Size (KByte) 4096 Total PE 2500 Free PE 1220 Allocated PE 1280 PV UUID Es9jwb-IjiL-jtd5-TgBx-XSxK-Xshj-Wxnjni --- Physical Segments --- Physical extent 0 to 1279: Logical volume /dev/test-volume/LV0 Logical extents 0 to 1279 Physical extent 1280 to 2499: FREE

 

In Listing 11, note that there are 2,499-1,280 = 1,219 free contiguous extents available, meaning that we can move up to 1,219 extents from another PV to /dev/hda7.

If you want to free a PV for replacement purposes, it's a good idea to disable its allocation so that you can be sure it remains free until you remove it from the volume group. Issue this before moving out the data:

Listing 12. Before freeing, disable a PV's allocation
 

#Disable /dev/hda6 allocation pvchange -xn /dev/hda6

 

Once free, see that the PV /dev/hda6 is 1,200 extents large and there are no free extents. To move the data from this PV, issue the following:

Listing 13. Moving data from the freed PV
 

#Move allocated extents out of /dev/hda6 pvmove -i 10 /dev/hda6

 

The -i 10 parameter in Listing 13 tells pvmove to report back status once every 10 seconds. Depending on how large the data to be moved is, this operation can take several minutes (or hours). This can also be done in the background with the -b parameter. In this case, status would be reported to the syslog.

In case you just don't have enough free contiguous extents to use in a pvmove operation, remember that you can always add one or more disks/partitions to a VG, thus adding a contiguous space, free for pvmove use.

Other useful LVM operations
Consult the man pages for more details on these other useful LVM operations:

• pvresize extends PVs if the underlying partition has also been extended; it shrinks PV if the allocation map permits it.
• pvremove destroys PVs (wipes its metadata clean). Use only after the PV had been removed from a VG with vgreduce.
• vgreduce removes unallocated PVs from a volume group, reducing the VG.
• vgmerge merges two different VGs into one. The target VG can be online!
• vgsplit splits a volume group.
• vgchange changes attributes and permissions of a VG.
• lvchange changes attributes and permissions of a LV.
• lvconvert converts between a linear volume and a mirror or snapshot and vice versa.

Make backups with Snapshots

A consistent backup is achieved when no data is changed between the start and the end of the backup process. This can be hard to guarantee without stopping the system for the time required by the copy process.

Linux LVM implements a feature called Snapshots that does exactly what the name says: It's like taking a picture of a logical volume at a given moment in time. With a Snapshot, you are provided with two copies of the same LV—one can be used for backup purposes while the other continues in operation.

The two great advantages of Snapshots are:

1. Snapshot creation is instantaneous; no need to stop a production environment.
2. Two copies are made, but not at twice the size. A Snapshot will use only the space needed to accommodate the difference between the two LVs.

This is accomplished by having an exception list that is updated every time something changes between the LVs (formally known as CoW, Copy-on-Write).

Create a new Snapshot

In order to create a new Snapshot LV, use the same lvcreate command, specifying the -s parameter and an origin LV. The -L size in this case specifies the exception table size, which is how much difference the Snapshot will support before losing consistency.

Listing 14. Taking your first Snapshot
 

#create a Snapshot LV called 'snap' from origin LV 'test' lvcreate -s -L 2G -n snap/dev/test-volume/test

 

Use lvdisplay to query special information like CoW-size and CoW-usage:

Listing 15. How big and how used is your herd of CoWs?
 

lvdisplay /dev/vg00/snap --- Logical volume --- LV Name /dev/vg00/snap VG Name vg00 LV UUID QHVJYh-PR3s-A4SG-s4Aa-MyWN-Ra7a-HL47KL LV Write Access read/write LV snapshot status active destination for /dev/vg00/test LV Status available # open 0 LV Size 4.00 GB Current LE 1024 COW-table size 2.00 GB COW-table LE 512 Allocated to snapshot 54.16% Snapshot chunk size 8.00 KB Segments 1 Allocation inherit Read ahead sectors 0 Block device 254:5

 

Notice in Listing 15 that CoW is 2GB large, 54.16 % of which is already used.

For all intents and purposes, the Snapshot is a copy of the original LV. It can be mounted if a filesystem is present with:

#mount snapshot volume mount -o ro /dev/test-volume/test /mnt/snap

 

In this snippet code, the ro flag to mount it is read-only. You can make it read-only at the LVM level by appending a -p r to the lvcreate command.

Once the filesystem has been mounted, you can proceed with backup using tar, rsync, or whatever backup tool is desired. If the LV doesn't contain a filesystem, or if a raw backup is desired, it's also possible to use dd directly on the device node.

Once the copy process finishes and the Snapshot is no longer needed, simply unmount and scrap it using lvremove:

#remove snapshot lvremove /dev/test-volume/snap

 

For consistency, in case a database is on top of an LV and a consistent backup is desired, remember to flush tables and make the Snapshot volume while acquiring a read-lock (in this lovely sample pseudo-code):

SQL> flush tables read lock {create Snapshot} SQL> release read lock {start copy process from the snapshot LV}

 

Sample backup script

The script in Listing 16 is taken directly from my laptop where I make daily backups using rsync to a remote server. This is not intended for enterprise use—an incremental backup with history would make more sense there. The concept remains the same, though.

Listing 16. Simple sample backup script
 

#!/bin/sh # we need the dm-snapshot module modprobe dm-snapshot if [ -e /dev/vg00/home-snap ] then # remove left-overs, if any umount -f /mnt/home-snap && true lvremove -f /dev/vg00/home-snap fi # create snapshot, 1GB CoW space # that should be sufficient for accommodating changes during copy lvcreate -vs -p r -n home-snap -L 1G /dev/vg00/home mkdir -p /mnt/home-snap # mount recently-created snapshot as read-only mount -o ro /dev/vg00/home-snap /mnt/home-snap # magical rsync command__rsync -avhzPCi --delete -e "ssh -i /home/klausk/.ssh/id_rsa" \ --filter '- .Trash/' --filter '- *~' \ --filter '- .local/share/Trash/' \ --filter '- *.mp3' --filter '- *Cache*' --filter '- *cache*' \ /mnt/home-snap/klausk klausk2@pokgsa.ibm.comThis e-mail address is being protected from spam bots, you need JavaScript enabled to view it :bkp/ # unmount and scrap snapshot LV umount /mnt/home-snap lvremove -f /dev/vg00/home-snap

 

In special cases where the cycle can't be estimated or copy process times are long, a script could query the Snapshot CoW usage with lvdisplay and extend the LV on demand. In extreme cases, you could opt for a Snapshot the same size as the original LV—that way, changes can never be larger than the whole volume!

Recent LVM2 developments allow a logical volume to sport high-availability features by having two or more mirrors each which can be placed under different physical volumes (or different devices). dmeventd can bring a PV offline without service prejudice when an I/O error is detected in the device. Refer to lvcreate(8), lvconvert(8), and lvchange(8) man pages for more info.

For hardware that supports it, it's possible to use dm_multipath for using different channels to access a single device, having a fail-over possibility in case a channel goes down. Refer to the dm_multipath and multipathd documentation for more details.

Transparent device encryption

You can transparently encrypt a block device or a logical volume with dm_crypt. Refer to the dm_crypt documentation and the cryptsetup(8) man page for more info.

redhat.com The Linux Logical Volume Manager by Heinz Mauelshagen and Matthew O'Keefe

• Introduction
• Basic LVM commands
• Differences between LVM1 and LVM2
• Summary
• About the authors

Storage technology plays a critical role in increasing the performance, availability, and manageability of Linux servers. One of the most important new developments in the Linux 2.6 kernel—on which the Red Hat® Enterprise Linux® 4 kernel is based—is the Linux Logical Volume Manager, version 2 (or LVM 2). It combines a more consistent and robust internal design with important new features including volume mirroring and clustering, yet it is upwardly compatible with the original Logical Volume Manager 1 (LVM 1) commands and metadata. This article summarizes the basic principles behind the LVM and provide examples of basic operations to be performed with it.

Introduction

Logical volume management is a widely-used technique for deploying logical rather than physical storage. With LVM, "logical" partitions can span across physical hard drives and can be resized (unlike traditional ext3 "raw" partitions). A physical disk is divided into one or more physical volumes (Pvs), and logical volume groups (VGs) are created by combining PVs as shown in Figure 1. LVM internal organization. Notice the VGs can be an aggregate of PVs from multiple physical disks.

Figure 2. Mapping logical extents to physical extents shows how the logical volumes are mapped onto physical volumes. Each PV consists of a number of fixed-size physical extents (PEs); similarly, each LV consists of a number of fixed-size logical extents (LEs). (LEs and PEs are always the same size, the default in LVM 2 is 4 MB.) An LV is created by mapping logical extents to physical extents, so that references to logical block numbers are resolved to physical block numbers. These mappings can be constructed to achieve particular performance, scalability, or availability goals.

For example, multiple PVs can be connected together to create a single large logical volume as shown in Figure 3. LVM linear mapping. This approach, known as a linear mapping, allows a file system or database larger than a single volume to be created using two physical disks. An alternative approach is a striped mapping, in which stripes (groups of contiguous physical extents) from alternate PVs are mapped to a single LV, as shown in Figure 4. LVM striped mapping. The striped mapping allows a single logical volume to nearly achieve the combined performance of two PVs and is used quite often to achieve high-bandwidth disk transfers.

Figure 4. LVM striped mapping (4 physical extents per stripe)

Through these different types of logical-to-physical mappings, LVM can achieve four important advantages over raw physical partitions: