DF speed – Volume fill up speed – ddf & ddf-auto & ddf-auto-i script
Need to know how fast in Kilobytes per second or Megabytes per second you mount point or volume is filling up (or emptying). This script will inform you of just that.
Need to know the speed of a transfer, you can just measure how fast df’s used speed is changing for a certain volume and assume speed of transfer is analogous to the speed of the volume its filling up (of course there could be other things filling up or emptying the volume). This script can also be used to see how fast a volume is freeing space.
CATCH: not every copy of a file immediately updates the DF values. Seems that the sync function does it. But for those filesystems that update DF value reliably and continously this will work.
Volume is being filled up returns a positive speed value.
Volume being emptied, growing in free space, returns a negative speed value.
Its pretty simple algorithm, we just take df at one point in time, then at another and measure the difference. Since linux’s time resolution is only 1 second with “date +$s ” (which gives seconds since the 1970s epoch), we must specify a delta time/difference in measurements in seconds. So a good delta time is like 2 seconds, but you can pick 5 seconds.
There are 3 scripts: ddf and ddf-auto
ddf should be used in conjunction with a while loop or watch also it can be manually run (it will tell you the difference in time since last run)
ddf-auto has the while loop built in, you get to pick the delta time in the argument (so you need to select volume name and delta time). ddf-auto-i is used in the same manner as ddf-auto meaning you need to specify volume name and the delta time, it gives you more information like inodes used and inodes difference (ddf and ddf-auto dont look at inodes). Also ddf-auto-i measures difference with respect to the start point (current value MINUS start value). Where as ddf and ddf-auto are measuring speed changes instantenously (current value MINUS last value a second or two ago)
Usage and examples of use are listed in the comments. Just make sure you choose a unique name for the first argument which sets the “device or mount point”, it has to be unique and it has to match how its seen in df, because the script greps for values from df’s output.
DDF SCRIPT – I saved mine as /sbin/ddf
#!/bin/bash
# measure instantaenous change of used space in volume from df
# usage: ddf <device or mount point as seen in df>
# example1: run "ddf /mnt" then wait a few seconds and repeat the command
# example2: run "watch -n2 ddf /mnt" to get df differences at 2 seconds, you can change the interval by changing -n2 to anything else. The units are in seconds with watch
# example3: run "while true; do ddf /mnt; sleep 2; done" same results as above and its similar to running "ddf-auto /mnt 2"
DEVICEkvk=$1;
###########
# KEEP ONE OF THE SYNC123 COMMENTED - --sync option runs "sync" before showing df stats, which is useful for btrfs systems and the like, however if the system is large and many files sync might take a long time to complete (and it might appear hung, if thats the case just let it finish)
SYNC123="--sync" # DO SYNC OR NOT - UNCOMMENT THIS ONE IF YOU WANT TO DO SYNC (BEFORE EACH READING)
#SYNC123="" # DO SYNC OR NOT - UNCOMMENT THIS ONE IF YOU DONT WANT TO DO SYNC (BEFORE EACH READING)
###########
# ----GET NEW VALUES----
USEDkvk=`df $SYNC123 -P | grep $DEVICEkvk | awk '{print $3}'`;
TIMEkvk=`date +%s`;
# ----GRAB OLD VALUES---- note: didnt use export, because it doesnt work with "watch"
OLDUSEDkvk=`cat /tmp/ddf-oldused`;
OLDTIMEkvk=`cat /tmp/ddf-oldtime`;
# ----OUTPUT-----
echo "==== DEVMOUNT: $1 - `date` - USED: $USEDkvk kb===="
(df -P && df -Ph) | grep -e "$DEVICEkvk\|Used";
echo $USEDkvk $TIMEkvk $OLDUSEDkvk $OLDTIMEkvk | awk '{DT=$2-$4; SKB=($1-$3)/DT; SMB=SKB/1024; print "NEW_USED: " $1 " KiB - OLD_USED: " $3 " KiB\nNEW_TIME: " $2 " s - OLD_TIME: " $4 " - DELTA_TIME: " DT "s\n---- SPEED: " SKB " KiB/sec, " SMB " MiB/sec ---" }';
# ----STORE INTO VARIABLES----
echo $USEDkvk > /tmp/ddf-oldused
echo $TIMEkvk > /tmp/ddf-oldtime
DDF-AUTO SCRIPT – I saved mine as /sbin/ddf-auto
#!/bin/bash
#measure instantaneous speed of change of DF (volume space)
#usage: ddf-auto <device or mountpoint as seen in df> <interval in seconds>
#example: ddf-auto /mnt 5
DEVICEkvk=$1;
###########
# KEEP ONE OF THE SYNC123 COMMENTED - --sync option runs "sync" before showing df stats, which is useful for btrfs systems and the like, however if the system is large and many files sync might take a long time to complete (and it might appear hung, if thats the case just let it finish)
SYNC123="--sync" # DO SYNC OR NOT - UNCOMMENT THIS ONE IF YOU WANT TO DO SYNC (BEFORE EACH READING)
#SYNC123="" # DO SYNC OR NOT - UNCOMMENT THIS ONE IF YOU DONT WANT TO DO SYNC (BEFORE EACH READING)
###########
while true;
do
# ----GET NEW VALUES----
USEDkvk=`df $SYNC123 -P | grep $DEVICEkvk | awk '{print $3}'`;
TIMEkvk=`date +%s`;
# ----OUTPUT-----
echo "##=== DEVMOUNT: $1 - `date` - USED: $USEDkvk kb===##"
(df -P && df -Ph) | grep -e "$DEVICEkvk\|Used";
echo $USEDkvk $TIMEkvk $OLDUSEDkvk $OLDTIMEkvk | awk '{DT=$2-$4; SKB=($1-$3)/DT; SMB=SKB/1024; print "NEW_USED: " $1 " KiB - OLD_USED: " $3 " KiB\nNEW_TIME: " $2 " s - OLD_TIME: " $4 " - DELTA_TIME: " DT "s\n---- SPEED: " SKB " KiB/sec, " SMB " MiB/sec ---" }';
# ----STORE INTO VARIABLES----
OLDUSEDkvk=$USEDkvk
OLDTIMEkvk=$TIMEkvk
sleep $2
done
DDF-AUTO-I – I saved mine as /sbin/ddf-auto-i
#!/bin/bash
#Shows difference and average speed of USED SPACE and INODE USAGE with respect to when you launch the program (now - initial) instead of (now - interval time ago)
#NOTE: systems like btrfs always show inode usage as 0 from df
#usage: ddf-auto-i <device or mountpoint as seen in df> <interval in seconds>
#example: ddf-auto-i /mnt 5
DEVICEkvk=$1;
###########
# KEEP ONE OF THE SYNC123 COMMENTED - --sync option runs "sync" before showing df stats, which is useful for btrfs systems and the like, however if the system is large and many files sync might take a long time to complete (and it might appear hung, if thats the case just let it finish)
SYNC123="--sync" # DO SYNC OR NOT - UNCOMMENT THIS ONE IF YOU WANT TO DO SYNC (BEFORE EACH READING)
#SYNC123="" # DO SYNC OR NOT - UNCOMMENT THIS ONE IF YOU DONT WANT TO DO SYNC (BEFORE EACH READING)
###########
# ----GET INIT VALUES----
USED0kvk=`df $SYNC123 -P | grep $DEVICEkvk | awk '{print $3}'`;
IUSED0kvk=`df -Pi | grep $DEVICEkvk | awk '{print $3}'`;
TIME0kvk=`date +%s`;
DATE0kvk=`date`;
echo "WELCOME TO DDF-AUTO-I"
echo "#####################"
echo "---- Ignore first result as it always errors out ----"
echo "---- USED and INODES are current values, USED0 and INODES0 are init values @ start time ----"
echo "---- INODES can be approximated to number of files and folders essentially ----"
echo "---- dINODES can be analagous to the number of files and folders transfered/created ----"
echo "---- dUSED is how much larger or smaller the Filesystem is ----"
echo "---- Positive dINODES and dUSED means space & inodes are being used up (ex: file transfer)----"
echo "---- Negative dINODES and dUSED means space & inodes are being freed up (ex: delete operation)----"
echo "---- All units are base2, ex: 1 KiB = 1024 Bytes and 1 MiB = 1024 * 1024 Bytes ----"
while true;
do
echo
# ----GET NEW VALUES----
USEDkvk=`df $SYNC123 -P | grep $DEVICEkvk | awk '{print $3}'`;
IUSEDkvk=`df -Pi | grep $DEVICEkvk | awk '{print $3}'`;
TIMEkvk=`date +%s`;
# ----OUTPUT-----
echo "#####== `date`, MNT $1, USED: "$USEDkvk" KiB, INODES: $IUSEDkvk ==####"
echo "TIME0 @ $DATE0kvk"
(df -P) | grep -e "$DEVICEkvk\|Used" && (df -Ph) | grep -e "$DEVICEkvk" && df -Pi | grep -e "$DEVICEkvk\|Used";
echo $USEDkvk $TIMEkvk $USED0kvk $TIME0kvk | awk '{DT=($2-$4); DUSED=($1-$3); SKB=DUSED/DT; SMB=SKB/1024; print "USED " $1 " KiB, USED0 " $3 " KiB **** dUSED " DUSED " KiB, " DUSED/1024 " MiB, " DUSED/1024/1024"GiB ****\nTIME " $2 " s, TIME0 " $4 " s, dT " DT " s\n-- SPD: " SKB " KiB/sec, " SMB " MiB/sec --" }';
echo $IUSEDkvk $TIMEkvk $IUSED0kvk $TIME0kvk | awk '{DT=$2-$4; DINODE=($1-$3);ISPD=DINODE/DT; print "INODES " $1 " - INODES0 " $3 " **** dINODES " DINODE " - RATE " ISPD " inodes/sec ****";}';
sleep $2
done
Example:
I have an rsync job running from 1 volume to another doing a backup. I want to know the overall transfer speed (rsync only shows progress and speed per file). My destination volume is getting filled up and not the source, so its size will be changing – you have to identify which volume will have the changing size. You can do so by running “df” a couple times and notice which volume is changing. In my case I see that the volume /dev/mapper/vo-both (which is mounted to /mnt/both) is changing – and obviously I know this as thats where I instructed my rsync job to copy to.
The first argument with ddf and ddf-auto is the mount point or volume name as seen from df. The catch is that you have to select a unique name for the argument.
Running df, I see the following:
Filesystem 1K-blocks Used Available Use% Mounted on /dev/md127 100000 15000 85000 15% /mnt/both/md127/ /dev/mapper/vo-both 2580272 698884 1750316 29% /mnt/both
I want to monitor vo-both which is mounted to /mnt/both. So I could select “/dev/mapper/vo-both” for the first argument. Notice that I can’t select the mount point in this case because the mount point “/mnt/both” is also shared with the above mount point “/mnt/both/md127” and I dont want to monitor that devices df speed. You have to think of what grep will find, the mission is for grep to only find 1 entry. So If I select /mnt/both it will find 2 lines. However if I select /dev/mapper/vo-both it will find that 1 entry that I need. Notice that I can also just type “vo-both” because its still unique and it will only grep out the last line (and nothing extra).
– Meaning this will work (only 1 line matches the 1st argument in df): # /sbin/ddf /dev/mapper/vo-both
– This will also work (only 1 line matches the 1st argument in df): # /sbin/ddf vo-both
– But this will not work (more then 1 line matching in df): # /sbin/ddf /mnt/both
– This will not work as well (more then 1 line matching in df): # /sbin/ddf both
– Technically I can also use 2580272, because that is the total space of the volume and that will never change (unless I shrink or enlarge the volume) – this will work – however its not human readable: # /sbin/ddf 2580272
So I will pick vo-both for the first argument so that I can monitor the device /dev/mapper/vo-both fill up speed (df speed).
For ddf I dont need to specify a time interval in the ddf arguments, but I do within the watch. I give it a 2 second interval with -n2 (this is watches argument)
# /sbin/ddf vo-both
Again this would give me same results:
# /sbin/ddf /dev/mapper/vo-both
For ddf-auto I have to specify a time interval in the ddf-auto arguments (its the second argument), so I select 2 seconds, just by typing 2 for that argument. The first argument will be the same as the first argument on ddf.
# /sbin/ddf-auto vo-both 2
Again this would give me same results:
# /sbin/ddf-auto /dev/mapper/vo-both 2
NOTE: I put my ddf and ddf-auto in /sbin as I will be using them quiet often (I could of just put it in /usr/local/bin, but I was lazy and /sbin was shorter to type, and I didnt think of /bin at the time, which also would of worked)
Example output of ddf:
# watch -n2 /sbin/ddf vo-both #OUTPUT - screen refreshes every 2 seconds, with updated output: Every 2.0s: /sbin/ddf vo-both Sun Apr 27 05:51:26 2014 ==== DEVMOUNT: vo-both - Sun Apr 27 05:51:27 PDT 2014 - USED: 696904 kb==== Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vo-both 2580272 698884 1750316 29% /mnt/both Filesystem Size Used Avail Use% Mounted on /dev/mapper/vo-both 2.5G 683M 1.7G 29% /mnt/both 696904 1398603087 207388 1398602228 NEW_USED: 696904 KiB - OLD_USED: 207388 KiB NEW_TIME: 1398603087 s - OLD_TIME: 1398602228 - DELTA_TIME: 859s ---- SPEED: 569.867 KiB/sec, 0.556511 MiB/sec ---
Example output ddf-auto:
# /sbin/ddf vo-both 2 # OUTPUT - scrolls down the screen, with new info every 2 seconds: ==== DEVMOUNT: vo-both - Sun Apr 27 05:51:03 PDT 2014 - USED: 619896 kb==== Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vo-both 2580272 619896 1829304 26% /mnt/both Filesystem Size Used Avail Use% Mounted on /dev/mapper/vo-both 2.5G 606M 1.8G 26% /mnt/both 619896 1398603063 610524 1398603061 NEW_USED: 619896 KiB - OLD_USED: 610524 KiB NEW_TIME: 1398603063 s - OLD_TIME: 1398603061 - DELTA_TIME: 2s ---- SPEED: 4686 KiB/sec, 4.57617 MiB/sec --- ==== DEVMOUNT: vo-both - Sun Apr 27 05:51:05 PDT 2014 - USED: 629896 kb==== Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vo-both 2580272 629936 1819264 26% /mnt/both Filesystem Size Used Avail Use% Mounted on /dev/mapper/vo-both 2.5G 616M 1.8G 26% /mnt/both 629896 1398603065 619896 1398603063 NEW_USED: 629896 KiB - OLD_USED: 619896 KiB NEW_TIME: 1398603065 s - OLD_TIME: 1398603063 - DELTA_TIME: 2s ---- SPEED: 5000 KiB/sec, 4.88281 MiB/sec ---
Example output ddf-auto-i:
# /sbin/ddf-auto-i vo-both 5 # after launching the above command, I then immediately start my transfer (or clean up procedure if thats what your doing) so that I can see the difference caused by the procedure. # OUTPUT: ignore the awk errors in the beginning (the first measurement will always produce that error as there is 0 difference and we are dividing by zero, every measurement after that is legit though) ##=== DEVMOUNT: vo-both, Sun Apr 27 20:26:21 PDT 2014, USED: 73220 kb, INODES: 10 ===## Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vo-both 2580272 73220 2375980 3% /mnt/both Filesystem Size Used Avail Use% Mounted on /dev/mapper/vo-both 2.5G 72M 2.3G 3% /mnt/both Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vo-both 163840 10 163830 1% /mnt/both awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted ##=== DEVMOUNT: vo-both, Sun Apr 27 20:26:26 PDT 2014, USED: 154544 kb, INODES: 293 ===## Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vo-both 2580272 154584 2294616 7% /mnt/both Filesystem Size Used Avail Use% Mounted on /dev/mapper/vo-both 2.5G 151M 2.2G 7% /mnt/both Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vo-both 163840 293 163547 1% /mnt/both USED_NOW: 154544 KiB - USED_INIT: 73220 KiB - D_USED: 81324 KiB, 79.418 MiB, 0.0775566 GiB TIME_NOW: 1398655586 s - TIME_INIT: 1398655581 - DELTA_TIME: 5s ---- SPEED: 16264.8 KiB/sec, 15.8836 MiB/sec --- INODES_NOW: 293 - INODES_INIT: 10 - DELTA_INODES: 283 ---- INODES_SPEED: 56.6 inodes/sec ##=== DEVMOUNT: vo-both, Sun Apr 27 20:26:32 PDT 2014, USED: 168384 kb, INODES: 293 ===## Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/vo-both 2580272 168384 2280816 7% /mnt/both Filesystem Size Used Avail Use% Mounted on /dev/mapper/vo-both 2.5G 165M 2.2G 7% /mnt/both Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vo-both 163840 293 163547 1% /mnt/both USED_NOW: 168384 KiB - USED_INIT: 73220 KiB - D_USED: 95164 KiB, 92.9336 MiB, 0.0907555 GiB TIME_NOW: 1398655592 s - TIME_INIT: 1398655581 - DELTA_TIME: 11s ---- SPEED: 8651.27 KiB/sec, 8.44851 MiB/sec --- INODES_NOW: 293 - INODES_INIT: 10 - DELTA_INODES: 283 ---- INODES_SPEED: 25.7273 inodes/sec