Every once in a while, you may find that the file system you are using is full. Usually this will manifest with a message like:
When this happens, check if the file system is under Logical Volume Management or LVM. You can do this a number of ways, but a simple way is with the lvs command. If the file system is NOT under LVM you will see something like:
But if the file system is under LVM, you will see something like:
/home/grokman % sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home brackin-deb-1-vg -wi-ao---- 32.00g
root brackin-deb-1-vg -wi-ao---- <18.38g
swap_1 brackin-deb-1-vg -wi-ao---- <4.00g
tmp brackin-deb-1-vg -wi-ao---- 380.00m
var brackin-deb-1-vg -wi-ao---- <5.12g If your file system is under LVM and you have run out of space on one file system, you may be able to re-distribute the available space under LVM to solve your problem. In this post, I am going to show an example of how that can be done.
In this example, the /home file system is 100% full and needs more space:
grokman@testbox1:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 932K 1.6G 1% /run
/dev/sda2 2.8G 880M 1.8G 34% /
/dev/mapper/vg-usr 8.4G 2.2G 5.9G 27% /usr
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/mapper/vg-var 8.4G 971M 7.1G 12% /var
/dev/sda1 464M 106M 331M 25% /boot
/dev/mapper/vg-opt 3.4G 193M 3.0G 6% /opt
/dev/mapper/vg-home 1.9G 1.8G 0 100% /home
/dev/mapper/vg-tmp 3.4G 48M 3.1G 2% /tmp
/dev/mapper/vg-var_log 3.4G 507M 2.7G 16% /var/log
/dev/mapper/vg-var_log_audit 945M 4.5M 876M 1% /var/log/audit
tmpfs 1.6G 0 1.6G 0% /run/user/1000
So, to solve this problem, we could mount another physical volume (PV) and get more space from there, or we can find a logical volume (LV) that has space to spare. One thing about this… in order to shrink an LV, we need to un-mount it. And, if you want to be able to just do this from an SSH session, you only want to choose a volume to get the space from that is is not very busy. I can see /var has some extra space, but I also know /var is typically very a busy file system (even though, in this case, /var/log is created as a separate LV from /var, I don’t want to take chances). I probably don’t want to bother with that one. Instead I could choose from one that is perhaps less busy like /opt. One way to see if any processes are running from there would be:
grokman@testbox1:~$ ps -ef |grep opt
jenkins 12103 9781 0 11:12 pts/1 00:00:00 grep --color=auto optSo then now I can see not much is going on with the /opt file system and it can be considered as a candidate.
grokman@testbox1:~$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home vg -wi-ao---- <1.91g
opt vg -wi-ao---- <3.45g
swap vg -wi-ao---- 3.81g
tmp vg -wi-ao---- <3.45g
usr vg -wi-ao---- 8.66g
var vg -wi-ao---- 8.66g
var_log vg -wi-ao---- <3.45g
var_log_audit vg -wi-ao---- 976.00m In this case, I decide to take 1.1GB from /opt and give it to /home. Let’s see how this is done.
First, we will un-mount /opt:
grokman@testbox1:~$ sudo umount /opt
grokman@testbox1:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 932K 1.6G 1% /run
/dev/sda2 2.8G 880M 1.8G 34% /
/dev/mapper/vg-usr 8.4G 2.2G 5.9G 27% /usr
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/mapper/vg-var 8.4G 971M 7.1G 12% /var
/dev/sda1 464M 106M 331M 25% /boot
/dev/mapper/vg-home 1.9G 1.8G 0 100% /home
/dev/mapper/vg-var_log 3.4G 507M 2.7G 16% /var/log
/dev/mapper/vg-tmp 3.4G 48M 3.1G 2% /tmp
/dev/mapper/vg-var_log_audit 945M 4.5M 876M 1% /var/log/audit
tmpfs 1.6G 0 1.6G 0% /run/user/1000Once we have the source file system un-mounted, its a good chance to check it for any issues:
grokman@testbox1:~$ sudo e2fsck -ff /dev/mapper/vg-opt
e2fsck 1.44.1 (24-Mar-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/vg-opt: 812/225792 files (0.1% non-contiguous), 79821/903168 blocksWith everything checking out ok there, we can commence to shrinking the source file system and LV. Now, it is possible to do this in 2 separate steps, but to minimize mistakes and miscalculations, you can do both steps with one command like so:
grokman@testbox1:~$ sudo lvreduce --resizefs -L 2300M /dev/mapper/vg-opt
fsck from util-linux 2.31.1
/dev/mapper/vg-opt: clean, 812/225792 files, 79821/903168 blocks
resize2fs 1.44.1 (24-Mar-2018)
Resizing the filesystem on /dev/mapper/vg-opt to 588800 (4k) blocks.
The filesystem on /dev/mapper/vg-opt is now 588800 (4k) blocks long.
Size of logical volume vg/opt changed from <3.45 GiB (882 extents) to <2.25 GiB (575 extents).
Logical volume vg/opt successfully resized.
grokman@testbox1:~$ echo $?
0
Here I am reducing /opt down from 3.45GB to 2.25GB. Note that you can’t use decimal points with these commands. The ‘echo $?’ command shows the status of the last run command & of course, 0 means success.
A note about the “L” argument: This argument can be a bit confusing. “-L” is basically shorthand for “logical size”, and the number that follows it will determine the new size according to:
- number not preceded by anything (eg 2300M) – this will be the new size
- number preceded by a “+” sign (eg +1GB) – the new size will increase by this number
- number preceded by a “-” sign (eg -1GB) – the new size will shrink by this number
At this point, we can mount /opt & check its size:
grokman@testbox1:~$ sudo mount /dev/mapper/vg-opt /opt
grokman@testbox1:~$ df -h /dev/mapper/vg-opt
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg-opt 2.2G 191M 1.9G 10% /optNext, we can take some, or all of the newly available space we now have, and allocate it to the LV & then the file system for /home. Since we reduced from 3.45 – 2.25 = 1.2GB, we can add up to about 1200MB. Here I will add 1100MB:
grokman@testbox1:~$ sudo lvextend -L +1100M /dev/mapper/vg-home
Size of logical volume vg/home changed from <1.91 GiB (488 extents) to 2.98 GiB (763 extents).
Logical volume vg/home successfully resized.
grokman@testbox:~$ sudo resize2fs /dev/mapper/vg-home
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/mapper/vg-home is mounted on /home; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/vg-home is now 781312 (4k) blocks long.
grokman@testbox1:~$ echo $?
0Finally, we can check that we now we have enough space on the target file system /home:
grokman@testbox1:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 932K 1.6G 1% /run
/dev/sda2 2.8G 880M 1.8G 34% /
/dev/mapper/vg-usr 8.4G 2.2G 5.9G 27% /usr
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/mapper/vg-var 8.4G 971M 7.1G 12% /var
/dev/sda1 464M 106M 331M 25% /boot
/dev/mapper/vg-home 3.0G 1.8G 1.1G 63% /home
/dev/mapper/vg-var_log 3.4G 481M 2.7G 15% /var/log
/dev/mapper/vg-tmp 3.4G 48M 3.1G 2% /tmp
/dev/mapper/vg-var_log_audit 945M 4.5M 876M 1% /var/log/audit
tmpfs 1.6G 0 1.6G 0% /run/user/1000
/dev/mapper/vg-opt 2.2G 191M 1.9G 10% /opHappy resizing!
Tested on: Ubuntu 18.04