Inode Exhaustion

An inode is a filesystem record that stores a file's metadata. On filesystems like ext4, the number of inodes is fixed when the filesystem is created. When all are used, creating any file fails with No space left on device, even if df -h shows plenty of free gigabytes.

Why it matters

It causes confusing outages: sessions cannot be written, uploads fail, logs stop, mail cannot be queued and databases error out, while disk usage graphs look fine. The usual culprits are directories filled with millions of tiny files, such as PHP session files, cache entries, mail queue files or unrotated tiny logs.

Example

Compare space and inode usage:

df -h /var       # block usage
df -i /var       # inode usage: look for IUse% at 100%

Find the directory that holds the most files:

for d in /var/*; do echo "$(find "$d" -xdev | wc -l) $d"; done | sort -n | tail

Delete or archive the offending files, then fix the source with session or cache cleanup jobs. Some filesystems, such as XFS and Btrfs, allocate inodes dynamically, so this is mostly an ext-family problem.

Common mistakes

• Looking only at df -h and concluding the disk is fine.
• Deleting large files, which frees space but not inodes.
• Not alerting on inode percentage.

Related

• Saturation
• USE method
• Single point of failure
• Tool: Uptime monitor

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.