Slow space reclamation on Netapp NFS with Linux, different space utilization reported by du and df

‘du’ and ‘df’ tools report different space utilization (Oracle Doc ID 457444.1)

‘du’ and ‘df’ tools report different space utilization (Doc ID 457444.1) To BottomTo Bottom
Modified:May 13, 2011Type:PROBLEM
Rate this document Email link to this document Open document in new window Printable Page

In this Document
Symptoms
Cause
Solution

Applies to:
Linux OS – Version: Oracle Linux 4.4 to Oracle Linux 5.3 – Release: OL4U4 to OL5U3
Linux x86
Linux x86-64
Linux Kernel – Version: 4.4 to 5.3
Symptoms
The ‘du’ (/usr/bin/du) and ‘df’ (/bin/df) command output displays conflicting space utilisation values, for example:
# df -k /
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda6 9288792 8672768 144120 99% /

# du -xsh /
2.1G /

In the example above, ‘df’ reports 8.6 Gb to have been used on the root (/) filesystem, whereas ‘du’ reports only 2.1 Gb to have been used.

Cause
The ‘df’ command reports how many disk blocks are used, whilst ‘du’ traverses the filesystem and reports the actual number of blocks used (directory by directory), including any relating to files in use by processes.

In most cases, space utilisation values returned from ‘df’ and ‘du’ will be consistent. However, the potential exists for a running process to delete a large file, say. In this instance, according to ‘du’, the large file no longer exists, so the blocks associated with the deleted file are not reported. With the process still running, and with an open file descriptor still held against the deleted file, ‘df’ continues to track and report all disk blocks used, including those associated with the deleted (phantom) file. In this situation, the disk space associated with the deleted file will only be relinquished back to the system when the process completely releases the deleted file’s descriptor or the process terminates (either gracefully or killed).

Solution
The solution is to identify and stop (or kill) the process that continues to hold a file descriptor open for the deleted file. To do so, run the lsof command (/usr/sbin/lsof | grep deleted) as root to identify the holding process, for example:

# lsof | grep deleted
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cannaserv 3825 canna 0u CHR 136,0 2 /dev/pts/0 (deleted)
vmware 4295 root 6u REG 253,0 6770 13074503 /tmp/vmware-root/ui-4295.log (deleted)
vmware-re 4316 root 6u REG 253,0 6770 13074503 /tmp/vmware-root/ui-4295.log (deleted)
vmnet-nat 4448 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
vmware-se 4454 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
gdm-binar 4506 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
gconfd-2 5392 root 12wW REG 253,0 609 13090818 /tmp/gconfd-root/lock/0t1188207163ut519551u0p5392r346479926k3219784492 (deleted)
vmware-vm 5822 root 57u REG 253,0 6520832 13074477 /tmp/vmware-root/ram0 (deleted)
vmware-vm 16487 root 57u REG 253,0 11153408 13074520 /tmp/vmware-root/ram0 (deleted)
kdeinit 17991 root 17u REG 253,0 26712 13074524 /tmp/kde-root/khtmlcacheM7jXYb.tmp (deleted)
kdeinit 17991 root 18u REG 253,0 5631 13074501 /tmp/kde-root/khtmlcacheZlJmda.tmp (deleted)
kdeinit 17991 root 21u REG 253,0 44718 13074514 /tmp/kde-root/khtmlcacheH5m4lc.tmp (deleted)

In the example above, the 7th column in the output denotes the size of deleted files (in bytes). The 9th column denotes which file remains held open. The 1st and second columns denotes the process and pid that still holds the open file descriptor.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s