View source for ENOSPC error, yet 'df' shows available space

[[Category:Engineering]]

You are getting errors of '''ENOSPC (No space left on device)''' when you try to create new files, but running '''df''' shows plenty of space free. You may also notice that log files continue to grow. How can they grow if the system keeps giving out of disk space errors? This all happens even if you are root.

<pre>
# pwd  
/root

# df /root
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_main-lv_boot
                     195249148  83551804 109744252  44% /

# touch foo
touch: cannot touch `foo': No space left on device

# df -T /root
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_main-lv_boot
              ext4   195249148  83551804 109744252  44% /

# df -Th /root
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/vg_main-lv_boot
              ext4    187G   80G  105G  44% /
</pre>

The trick is to look at how many '''inodes''' are free:
<pre>
# df -Ti /root
Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_main-lv_boot
              ext4    190848  190848       0  100% /
</pre>

Ooops! Plenty of blocks free, but not enough inodes.