Python 101 – Launching Subprocesses with Python

There are times when you are writing an application and you need to run another application. For example, you may need to open Microsoft Notepad on Windows for some reason. Or if you are on Linux, you might want to run grep. Python has support for launching external applications via the subprocess module.

The subprocess module has been a part of Python since Python 2.4. Before that you needed to use the os module. You will find that the subprocess module is quite capable and straightforward to use.

In this article you will learn how to use:

• The subprocess.run() Function
• The subprocess.Popen() Class
• The subprocess.Popen.communicate() Function
• Reading and Writing with stdin and stdout

Let’s get started!

The subprocess.run() Function

The run() function was added in Python 3.5. The run() function is the recommended method of using subprocess.

It can often be generally helpful to look at the definition of a function, to better understand how it works:

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None,
    capture_output=False, shell=False, cwd=None, timeout=None, check=False, 
    encoding=None, errors=None, text=None, env=None, universal_newlines=None)

You do not need to know what all of these arguments do to use run() effectively. In fact, most of the time you can probably get away with only knowing what goes in as the first argument and whether or not to enable shell. The rest of the arguments are helpful for very specific use-cases.

Let’s try running a common Linux / Mac command, ls. The ls command is used to list the files in a directory. By default, it will list the files in the directory you are currently in.

To run it with subprocess, you would do the following:

>>> import subprocess
>>> subprocess.run(['ls'])
filename
CompletedProcess(args=['ls'], returncode=0)

You can also set shell=True, which will run the command through the shell itself. Most of the time, you will not need to do this, but can be useful if you need more control over the process and want to access shell pipes and wildcards.

But what if you want to keep the output from a command so you can use it later on? Let’s find out how you would do that next!

Getting the Output

Quite often you will want to get the output from an external process and then do something with that data. To get output from run() you can set the capture_output argument to True:

>>> subprocess.run(['ls', '-l'], capture_output=True)
CompletedProcess(args=['ls', '-l'], returncode=0, 
    stdout=b'total 40\n-rw-r--r--@ 1 michael  staff  17083 Apr 15 13:17 some_file\n', 
    stderr=b'')

Now this isn’t too helpful as you didn’t save the returned output to a variable. Go ahead and update the code so that you do and then you’ll be able to access stdout.

 >>> output = subprocess.run(['ls', '-l'], capture_output=True)
>>> output.stdout
b'total 40\n-rw-r--r--@ 1 michael  staff  17083 Apr 15 13:17 some_file\n'

The output is a CompletedProcess class instance, which lets you access the args that you passed in, the returncode as well as stdout and stderr.

You will learn about the returncode in a moment. The stderr is where most programs print their error messages to, while stdout is for informational messages.

If you are interested, you can play around with this code and discover what if currently in those attributes, if anything:

output = subprocess.run(['ls', '-l'], capture_output=True)
print(output.returncode)
print(output.stdout)
print(out.stderr)

Let’s move on and learn about Popen next.

The subprocess.Popen() Class

The subprocess.Popen() class has been around since the subprocess module itself was added. It has been updated several times in Python 3. If you are interested in learning about some of those changes, you can read about them here:

• https://docs.python.org/3/library/subprocess.html#popen-constructor

You can think of Popen as the low-level version of run(). If you have an unusual use-case that run() cannot handle, then you should be using Popen instead.

For now, let’s look at how you would run the command in the previous section with Popen:

>>> import subprocess
>>> subprocess.Popen(['ls', '-l'])
<subprocess.Popen object at 0x10f88bdf0>
>>> total 40
-rw-r--r--@ 1 michael  staff  17083 Apr 15 13:17 some_file

>>>

The syntax is almost identical except that you are using Popen instead of run().

Here is how you might get the return code from the external process:

>>> process = subprocess.Popen(['ls', '-l'])
>>> total 40
-rw-r--r--@ 1 michael  staff  17083 Apr 15 13:17 some_file

>>> return_code = process.wait()
>>> return_code
0
>>>

A return_code of 0 means that the program finished successfully. If you open up a program with a user interface, such as Microsoft Notepad, you will need to switch back to your REPL or IDLE session to add the process.wait() line. The reason for this is that Notepad will appear over the top of your program.

If you do not add the process.wait() call to your script, then you won’t be able to catch the return code after manually closing any user interface program you may have started up via subprocess.

You can use your process handle to access the process id via the pid attribute. You can also kill (SIGKILL) the process by calling process.kill() or terminate (SIGTERM) it via process.terminate().

The subprocess.Popen.communicate() Function

There are times when you need to communicate with the process that you have spawned. You can use the Popen.communicate() method to send data to the process as well as extract data.

For this section, you will only use communicate() to extract data. Let’s use communicate() to get information using the ifconfig command, which you can use to get information about your computer’s network card on Linux or Mac. On Windows, you would use ipconfig. Note that there is a one-letter difference in this command, depending on your Operating System.

Here’s the code:

>>> import subprocess
>>> cmd = ['ifconfig']
>>> process = subprocess.Popen(cmd, 
                               stdout=subprocess.PIPE,
                               encoding='utf-8')
>>> data = process.communicate()
>>> print(data[0])
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
    options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
    inet 127.0.0.1 netmask 0xff000000 
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
    nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
XHC20: flags=0<> mtu 0
# -------- truncated --------

This code is set up a little differently than the last one. Let’s go over each piece in more detail.

The first thing to note is that you set the stdout parameter to a subprocess.PIPE. That allows you to capture anything that the process sends to stdout. You also set the encoding to utf-8. The reason you do that is to make the output a little easier to read, since the subprocess.Popen call returns bytes by default rather than strings.

The next step is to call communicate() which will capture the data from the process and return it. The communicate() method returns both stdout and stderr, so you will get a tuple. You didn’t capture stderr here, so that will be None.

Finally, you print out the data. The string is fairly long, so the output is truncated here.

Let’s move on and learn how you might read and write with subprocess!

Reading and Writing with stdin and stdout

Let’s pretend that your task for today is to write a Python program that checks the currently running processes on your Linux server and prints out the ones that are running with Python.

You can get a list of currently running processes using ps -ef. Normally you would use that command and “pipe” it to grep, another Linux command-line utility, for searching strings files.

Here is the complete Linux command you could use:

ps -ef | grep python

However, you want to translate that command into Python using the subprocess module.

Here is one way you can do that:

import subprocess

cmd = ['ps', '-ef']
ps = subprocess.Popen(cmd, stdout=subprocess.PIPE)

cmd = ['grep', 'python']
grep = subprocess.Popen(cmd, stdin=ps.stdout, stdout=subprocess.PIPE,
                        encoding='utf-8')

ps.stdout.close()
output, _ = grep.communicate()
python_processes = output.split('\n')
print(python_processes)

This code recreates the ps -ef command and uses subprocess.Popen to call it. You capture the output from the command using subprocess.PIPE. Then you also create the grep command.

For the grep command you set its stdin to be the output of the ps command. You also capture the stdout of the grep command and set the encoding to utf-8 as before.

This effectively gets the output from the ps command and “pipes” or feeds it into the grep command. Next, you close() the ps command’s stdout and use the grep command’s communicate() method to get output from grep.

To finish it up, you split the output on the newline (\n), which gives you a list of strings that should be a listing of all your active Python processes. If you don’t have any active Python processes running right now, the output will be an empty list.

You can always run ps -ef yourself and find something else to search for other than python and try that instead.

Wrapping Up

The subprocess module is quite versatile and gives you a rich interface to work with external processes.

In this article, you learned about:

• The subprocess.run() Function
• The subprocess.Popen() Class
• The subprocess.Popen.communicate() Function
• Reading and Writing with stdin and stdout

There is more to the subprocess module than what is covered here. However, you should now be able to use subprocess correctly. Go ahead and give it a try!