ENOSPC error, yet 'df' shows available space
You are getting errors of ENOSPC (No space left on device) when you try to create new files, but running df shows plenty of space free. You may also notice that log files continue to grow. How can they grow if the system keeps giving out of disk space errors? This all happens even if you are root.
# pwd /root # df /root Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_main-lv_boot 195249148 83551804 109744252 44% / # touch foo touch: cannot touch `foo': No space left on device # df -T /root Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_main-lv_boot ext4 195249148 83551804 109744252 44% / # df -Th /root Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_main-lv_boot ext4 187G 80G 105G 44% /
The trick is to look at how many inodes are free:
# df -Ti /root Filesystem Type Inodes IUsed IFree IUse% Mounted on /dev/mapper/vg_main-lv_boot ext4 190848 190848 0 100% /
Ooops! Plenty of blocks free, but not enough inodes. In my case, it turned out that I had setup an Ubuntu apt mirror on a server where the disk had been optimized for large files. I eventually tracked down the apt-mirror directory and deleted it.
# cd / # dircounts ... 13093 lib 31017 proc 57944 usr 113600 var # cd var # dircounts 1 crash 1 local 1 opt 1 tmp 3 lock 4 mail 19 backups 54 run 266 log 3652 lib 6293 www 14484 cache 88820 spool # cd spool # dircounts 1 mail 5 cron 8 exim4 64 postfix 88741 apt-mirror # df -i / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 190848 190848 0 100% / # rm -rf apt-mirror # df -i / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 190848 102415 88433 54% /
Notice that command dircounts. That's an alias that counts the number of files under each subdirectory of the current directory. I use that along with a dirsizes alias to hunt down space wasters. The dirsizes alias shows how many K bytes each subdirectory uses.
alias dirsizes='du -ks * | sort -n -b | column -t' alias dircounts='for dp in *; do [ -d $dp ] && echo $(find $dp|wc -l) $dp; done | sort -n -b | column -t'