How to Monitor the Progress of Linux Commands (With pv and progress)

Instead of flying blind, use the Linux pv and progress commands to track a command’s progress. These utilities will give you progress bars for commands that don’t normally have any. You’ll see an estimated time until completion, too.
If you are on a long haul flight on an aircraft without video screens in the seatbacks, it isn’t easy to know how far through your journey you are. You know when you took off. You know how long the flight is expected to take. But how do you know if you are on track, on time, or way behind schedule? If you don’t want to watch the in-flight movie you can usually switch your video screen to show a map with the position of your aircraft on it. You also get some statistics, such as an expected time of arrival (ETA), which is great.
Starting a command from the terminal window can sometimes feel like a long-haul flight without a video screen. You have nothing to give any indication if all is well or if the process has hung, nor how close to completion it is. A flashing cursor isn’t very informative.
The pv and progress commands give you some statistics and a little visual feedback. You can see how close the process is to complete. That means you get an ETA for your running processes. Compared with staring at a cursor, that wins hands down.

Installing pv

You must install pv.
To install pv on Ubuntu use this command:

sudo apt-get install pv

To install pv on Fedora use this command:

sudo dnf install pv

To install pv on Manjaro use this command:

sudo pacman -Syu pv

Using pv

pv stands for pipe viewer. Piping has to be involved in the command somewhere. Here’s an example where we’re piping an ISO image through zip to make a compressed zip file of the ISO.
To slow the commands down enough that a screenshot could be taken, some of the files in the examples used for this article were stored on an old, slow, external USB called SILVERXHD.

pv /media/dave/SILVERXHD/gparted-live-1.0.0-1-amd64.iso | zip > gparted.zip

The information pv gives us can be seen in the bottom line of the display.

From left to right, the information that is displayed is:

• The data transferred so far.
• The time elapsed fo far.
• The data transfer rate (throughput).
• A progress bar and a percentage completed figure.
• The estimated time left before completion (ETA).

Copying a File with pv

To copy a file with output from pv, use this command:

pv /media/dave/SILVERXHD/gparted-live-1.0.0-1-amd64.iso > gparted.iso

We get a progress report as the file is copied.

Copying Multiple Files with pv

To copy multiple files and folders with pv we need to use a little trick. We use tar to move the files for us.

tar -c help-files/ | pv | tar -x -C Documents/

The tar -c help-files/ portion of the command instructs tar to create (-c) an archive of the files in the help-files folder.  This is piped through pv so that we get a display of the progress. It is then piped back into tar for the last portion of the command. The archive is extracted (-x) and the directory is changed (-C) to Documents before the extraction.
So, the files and folders that are in help-files are copied to the Documents folder, with a progress display.

The output is slightly different this time.
We do not get an ETA. The progress bar now displays a moving indicator. It shows that the process is active, but it doesn’t grow from left to right like a traditional progress bar. pv is limited to displaying the information it can extract from the process that is being piped.

Using pv and tar to Create an Archive

Copying files with pv and tar does not leave us with an archive file. A sort of “virtual” archive is created by tar, which is fed straight back into tar to extract the files. If our objective is to copy files, that is accomplished. But what if we want to create an archive file?
We can still use tar to create an archive file and get a progress report from pv. The options used with tar are -c (create archive), -z (compress with gzip) and -f (filename of the archive).
Note that we’re using - as the filename, which causes tar to use stdout, and to write its output to the terminal window. We don’t see that output because it is piped through pv.
The actual name of the archive is going to be the filename that we pipe the output from pv into. In this case, it is “help-files.tgz”.

tar -czf - ./help-files/ | pv > help-files.tgz

We get the same progress indicators as before, and the archive file is created for us.

The pv Display Options

There are a number of options you can use with pv to change the details of its report.
If you use any of these options, all of the other options are turned off. So if you want to have three of the display options in use, then you need to specify those three options.
Using pv without any options is the same as using the -pterb options.

• -p: display the percentage complete. This is the progress bar and the percentage completed figure.
• -t: display the elapsed time.
• -e: display the ETA.
• -r: display the rate of data transfer.
• -b: display the byte count (data transferred so far).
• -n: display the percentage as an integer. This prints the percentage completed as an integer figure, with each new update on a new line.

Let’s repeat the last command and pass the -p (percentage completed) option to pv.

tar -czf - ./help-files/ | pv - p > help-files.tgz

This turns off all of the other display options. pv provides the percentage completed element only.
Because pv doesn’t get a percentage completed figure from tar, the progress bar is replaced with a moving indicator. There is no percentage figure.

Using pv With wc

We can use pv to pipe a  text file (or files) into wc. wc will then count the carriage returns, characters, and words and pv will give us a progress report.
Here we are piping all of the “.page” files in the help-files directory into wc.

When wc completes we can see our count of carriage returns (lines), characters and words from all of the “.page” files in the help-files folder.

Installing the progress Command

The progress command gives the same sort of useful information as pv, but it works with a specific set of Linux commands.
To install progress in Ubuntu, use this command:

sudo apt-get install progress

To install progress in Fedora, use this command:

sudo dnf install progress

To install progress in Manjaro, use this command:

sudo pacman -Syu progress

The Commands progress Works With

Typing progress in a terminal window and pressing Enter will give you a list of the commands that progress works with.

progress

Using Progress With Pipes

There are two techniques we can use to monitor commands with progress. The first is to use pipes.
The tar command is in the list of supported commands that progress can monitor, so let’s use tar.
The options we’ll use are the standard -c (create archive), -z (compress with gzip) and -f (filename) options. We’re going to create a compressed archive of everything in the help-files folder, and the archive will be named “help.tgz”.
We’re piping that into progress and using the -m (monitor) option so progress keeps reporting on the process until it has completed.

tar -czf help.tgz ./help-files/ | progress -m

The terminal window will show the progress of the tar command as it creates the archive.

As each file is processed, it is listed, with the following information:

• The process ID.
• The process name.
• Percentage completed.
• Data processed and total size of the file.
• Data rate (throughput).
• Estimated time remaining (ETA).

You might be surprised to see a second data set appear. This first data set is for tar. The second one is for gzip. tar calls gzip to perform the compression. Because gzip is in the list of supported commands, progress reports on it.

Using Progress in Continual Monitor Mode

You can use progress in a real-time continual monitor mode by using the -M (monitor) option.
Type the following command in a terminal window:

progress -M

progress will report that there are no commands running for it to monitor. But you are not returned to the command line. progress waits until a command that it can monitor starts. It will then automatically start reporting on it.

In another terminal window, type a command that is in the list of commands that progress can monitor.
We’re going to use cat. Commands that are over too quickly won’t register with progress, so we’ll list the contents of a very long text file.

cat words.page

In the terminal window with progress in it, you’ll see statistics for the cat command as it executes and works towards completion.

When cat finishes listing the file progress returns to its waiting state.
Each time one of the commands it can report on performs a sizeable task, progress will automatically monitor it and report on it.
That’s pretty neat.

100% Completed

Take the guesswork out of wondering how a long-running command is doing, and take a break from contemplating your cursor with pv and progress .