ENOSPC error, yet 'df' shows available space

From Noah.org
Jump to: navigation, search


You are getting errors of ENOSPC (No space left on device) when you try to create new files, but running df shows plenty of space free. You may also notice that log files continue to grow. How can they grow if the system keeps giving out of disk space errors? This all happens even if you are root.

# pwd  
/root

# df /root
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_main-lv_boot
                     195249148  83551804 109744252  44% /

# touch foo
touch: cannot touch `foo': No space left on device

# df -T /root
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_main-lv_boot
              ext4   195249148  83551804 109744252  44% /

# df -Th /root
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_boot
              ext4    187G   80G  105G  44% /

The trick is to look at how many inodes are free:

# df -Ti /root
Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_main-lv_boot
              ext4    190848  190848       0  100% /

Ooops! Plenty of blocks free, but not enough inodes. In my case, it turned out that I had setup an Ubuntu apt mirror on a server where the disk had been optimized for large files. I eventually tracked down the apt-mirror directory and deleted it.

# cd /

# dircounts
...
13093  lib
31017  proc
57944  usr
113600 var

# cd var

# dircounts
1      crash
1      local
1      opt
1      tmp
3      lock
4      mail
19     backups
54     run
266    log
3652   lib
6293   www
14484  cache
88820  spool

# cd spool

# dircounts
1      mail
5      cron
8      exim4
64     postfix
88741  apt-mirror

# df -i /
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             190848  190848       0  100% /

# rm -rf apt-mirror

# df -i /
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             190848  102415   88433   54% /

Notice that command dircounts. That's an alias that counts the number of files under each subdirectory of the current directory. I use that along with a dirsizes alias to hunt down space wasters. The dirsizes alias shows how many K bytes each subdirectory uses.

alias dirsizes='du -ks * | sort -n -b | column -t'
alias dircounts='for dp in *; do [ -d $dp ] && echo $(find $dp|wc -l) $dp; done | sort -n -b | column -t'