Optimizing NFS performance on Linux and Netapp

http://www.tldp.org/HOWTO/NFS-HOWTO/performance.html

Advertisements

Slow space reclamation on Netapp NFS with Linux, different space utilization reported by du and df

‘du’ and ‘df’ tools report different space utilization (Oracle Doc ID 457444.1)

‘du’ and ‘df’ tools report different space utilization (Doc ID 457444.1) To BottomTo Bottom
Modified:May 13, 2011Type:PROBLEM
Rate this document Email link to this document Open document in new window Printable Page

In this Document
Symptoms
Cause
Solution

Applies to:
Linux OS – Version: Oracle Linux 4.4 to Oracle Linux 5.3 – Release: OL4U4 to OL5U3
Linux x86
Linux x86-64
Linux Kernel – Version: 4.4 to 5.3
Symptoms
The ‘du’ (/usr/bin/du) and ‘df’ (/bin/df) command output displays conflicting space utilisation values, for example:
# df -k /
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda6 9288792 8672768 144120 99% /

# du -xsh /
2.1G /

In the example above, ‘df’ reports 8.6 Gb to have been used on the root (/) filesystem, whereas ‘du’ reports only 2.1 Gb to have been used.

Cause
The ‘df’ command reports how many disk blocks are used, whilst ‘du’ traverses the filesystem and reports the actual number of blocks used (directory by directory), including any relating to files in use by processes.

In most cases, space utilisation values returned from ‘df’ and ‘du’ will be consistent. However, the potential exists for a running process to delete a large file, say. In this instance, according to ‘du’, the large file no longer exists, so the blocks associated with the deleted file are not reported. With the process still running, and with an open file descriptor still held against the deleted file, ‘df’ continues to track and report all disk blocks used, including those associated with the deleted (phantom) file. In this situation, the disk space associated with the deleted file will only be relinquished back to the system when the process completely releases the deleted file’s descriptor or the process terminates (either gracefully or killed).

Solution
The solution is to identify and stop (or kill) the process that continues to hold a file descriptor open for the deleted file. To do so, run the lsof command (/usr/sbin/lsof | grep deleted) as root to identify the holding process, for example:

# lsof | grep deleted
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cannaserv 3825 canna 0u CHR 136,0 2 /dev/pts/0 (deleted)
vmware 4295 root 6u REG 253,0 6770 13074503 /tmp/vmware-root/ui-4295.log (deleted)
vmware-re 4316 root 6u REG 253,0 6770 13074503 /tmp/vmware-root/ui-4295.log (deleted)
vmnet-nat 4448 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
vmware-se 4454 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
gdm-binar 4506 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
gconfd-2 5392 root 12wW REG 253,0 609 13090818 /tmp/gconfd-root/lock/0t1188207163ut519551u0p5392r346479926k3219784492 (deleted)
vmware-vm 5822 root 57u REG 253,0 6520832 13074477 /tmp/vmware-root/ram0 (deleted)
vmware-vm 16487 root 57u REG 253,0 11153408 13074520 /tmp/vmware-root/ram0 (deleted)
kdeinit 17991 root 17u REG 253,0 26712 13074524 /tmp/kde-root/khtmlcacheM7jXYb.tmp (deleted)
kdeinit 17991 root 18u REG 253,0 5631 13074501 /tmp/kde-root/khtmlcacheZlJmda.tmp (deleted)
kdeinit 17991 root 21u REG 253,0 44718 13074514 /tmp/kde-root/khtmlcacheH5m4lc.tmp (deleted)

In the example above, the 7th column in the output denotes the size of deleted files (in bytes). The 9th column denotes which file remains held open. The 1st and second columns denotes the process and pid that still holds the open file descriptor.

slow space reclamation on Netapp NFS

Space reclamation is a background process. This normally is a slow process that will greatly consume resource on the filer. It would be recommended to perform space reclamation during the night time when there are less jobs running on the filer. As the Volumes are spanned within the Raid Groups of the Aggregates it will take WALF some time to free the blocks.

SnapDrive space reclamation in Windows Servers – Frequently Asked Question (FAQ)

https://kb.netapp.com/support/index?page=content&id=3011267

Space Reclamation: If Being Thin is Attractive, Then Thin Provisioning Makes for Really Sexy Storage

https://communities.netapp.com/community/netapp-blogs/sanbytes/blog/2012/05/29/space-reclamation-if-being-thin-is-attractive-then-thin-provisioning-makes-for-really-fine-storage

NetApp Thin-Provisioned LUNs on RHEL 6.2 Deployment Guide that outlines space reclamation best practices on Red Hat Linux
http://www.netapp.com/us/system/pdf-reader.aspx?m=tr-4046.pdf&cc=us