There are times when you are writing an application and you need to run another application. For example, you may need to open Microsoft Notepad on Windows for some reason. Or if you are on Linux, you might want to run grep. Python has support for launching external applications via the
subprocess module has been a part of Python since Python 2.4. Before that you needed to use the
os module. You will find that the
subprocess module is quite capable and straightforward to use.
In this article you will learn how to use:
- Reading and Writing with
Let’s get started!
run() function was added in Python 3.5. The
run() function is the recommended method of using
It can often be generally helpful to look at the definition of a function, to better understand how it works:
subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None)
You do not need to know what all of these arguments do to use
run() effectively. In fact, most of the time you can probably get away with only knowing what goes in as the first argument and whether or not to enable
shell. The rest of the arguments are helpful for very specific use-cases.
Let’s try running a common Linux / Mac command,
ls command is used to list the files in a directory. By default, it will list the files in the directory you are currently in.
To run it with
subprocess, you would do the following:
>>> import subprocess >>> subprocess.run(['ls']) filename CompletedProcess(args=['ls'], returncode=0)
You can also set
shell=True, which will run the command through the shell itself. Most of the time, you will not need to do this, but can be useful if you need more control over the process and want to access shell pipes and wildcards.
But what if you want to keep the output from a command so you can use it later on? Let’s find out how you would do that next!
Getting the Output
Quite often you will want to get the output from an external process and then do something with that data. To get output from
run() you can set the
capture_output argument to True:
>>> subprocess.run(['ls', '-l'], capture_output=True) CompletedProcess(args=['ls', '-l'], returncode=0, stdout=b'total 40\n-rw-r--r--@ 1 michael staff 17083 Apr 15 13:17 some_file\n', stderr=b'')
Now this isn’t too helpful as you didn’t save the returned output to a variable. Go ahead and update the code so that you do and then you’ll be able to access
>>> output = subprocess.run(['ls', '-l'], capture_output=True) >>> output.stdout b'total 40\n-rw-r--r--@ 1 michael staff 17083 Apr 15 13:17 some_file\n'
output is a
CompletedProcess class instance, which lets you access the
args that you passed in, the
returncode as well as
You will learn about the
returncode in a moment. The
stderr is where most programs print their error messages to, while
stdout is for informational messages.
If you are interested, you can play around with this code and discover what if currently in those attributes, if anything:
output = subprocess.run(['ls', '-l'], capture_output=True) print(output.returncode) print(output.stdout) print(out.stderr)
Let’s move on and learn about
subprocess.Popen() class has been around since the
subprocess module itself was added. It has been updated several times in Python 3. If you are interested in learning about some of those changes, you can read about them here:
You can think of
Popen as the low-level version of
run(). If you have an unusual use-case that
run() cannot handle, then you should be using
For now, let’s look at how you would run the command in the previous section with
>>> import subprocess >>> subprocess.Popen(['ls', '-l']) <subprocess.Popen object at 0x10f88bdf0> >>> total 40 -rw-r--r--@ 1 michael staff 17083 Apr 15 13:17 some_file >>>
The syntax is almost identical except that you are using
Popen instead of
Here is how you might get the return code from the external process:
>>> process = subprocess.Popen(['ls', '-l']) >>> total 40 -rw-r--r--@ 1 michael staff 17083 Apr 15 13:17 some_file >>> return_code = process.wait() >>> return_code 0 >>>
0 means that the program finished successfully. If you open up a program with a user interface, such as Microsoft Notepad, you will need to switch back to your REPL or IDLE session to add the
process.wait() line. The reason for this is that Notepad will appear over the top of your program.
If you do not add the
process.wait() call to your script, then you won’t be able to catch the return code after manually closing any user interface program you may have started up via
You can use your
process handle to access the process id via the
pid attribute. You can also kill (SIGKILL) the process by calling
process.kill() or terminate (SIGTERM) it via
There are times when you need to communicate with the process that you have spawned. You can use the
Popen.communicate() method to send data to the process as well as extract data.
For this section, you will only use
communicate() to extract data. Let’s use
communicate() to get information using the
ifconfig command, which you can use to get information about your computer’s network card on Linux or Mac. On Windows, you would use
ipconfig. Note that there is a one-letter difference in this command, depending on your Operating System.
Here’s the code:
>>> import subprocess >>> cmd = ['ifconfig'] >>> process = subprocess.Popen(cmd, stdout=subprocess.PIPE, encoding='utf-8') >>> data = process.communicate() >>> print(data) lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP> inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 nd6 options=201<PERFORMNUD,DAD> gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280 stf0: flags=0<> mtu 1280 XHC20: flags=0<> mtu 0 # -------- truncated --------
This code is set up a little differently than the last one. Let’s go over each piece in more detail.
The first thing to note is that you set the
stdout parameter to a
subprocess.PIPE. That allows you to capture anything that the process sends to
stdout. You also set the
utf-8. The reason you do that is to make the output a little easier to read, since the
subprocess.Popen call returns bytes by default rather than strings.
The next step is to call
communicate() which will capture the data from the process and return it. The
communicate() method returns both
stderr, so you will get a
tuple. You didn’t capture
stderr here, so that will be
Finally, you print out the data. The string is fairly long, so the output is truncated here.
Let’s move on and learn how you might read and write with
Reading and Writing with
Let’s pretend that your task for today is to write a Python program that checks the currently running processes on your Linux server and prints out the ones that are running with Python.
You can get a list of currently running processes using
ps -ef. Normally you would use that command and “pipe” it to
grep, another Linux command-line utility, for searching strings files.
Here is the complete Linux command you could use:
ps -ef | grep python
However, you want to translate that command into Python using the
Here is one way you can do that:
import subprocess cmd = ['ps', '-ef'] ps = subprocess.Popen(cmd, stdout=subprocess.PIPE) cmd = ['grep', 'python'] grep = subprocess.Popen(cmd, stdin=ps.stdout, stdout=subprocess.PIPE, encoding='utf-8') ps.stdout.close() output, _ = grep.communicate() python_processes = output.split('\n') print(python_processes)
This code recreates the
ps -ef command and uses
subprocess.Popen to call it. You capture the output from the command using
subprocess.PIPE. Then you also create the
grep command you set its
stdin to be the output of the
ps command. You also capture the
stdout of the
grep command and set the encoding to
utf-8 as before.
This effectively gets the output from the
ps command and “pipes” or feeds it into the
grep command. Next, you
stdout and use the
communicate() method to get output from
To finish it up, you split the output on the newline (
\n), which gives you a
list of strings that should be a listing of all your active Python processes. If you don’t have any active Python processes running right now, the output will be an empty list.
You can always run
ps -ef yourself and find something else to search for other than
python and try that instead.
subprocess module is quite versatile and gives you a rich interface to work with external processes.
In this article, you learned about:
- Reading and Writing with
There is more to the
subprocess module than what is covered here. However, you should now be able to use
subprocess correctly. Go ahead and give it a try!
2 thoughts on “Python 101 – Launching Subprocesses with Python”
Pingback: Python Monthly ???????? July 2020 | Zero To Mastery - INFOSHRI
Pingback: Python 101 - Documenting Your Code - Mouse Vs Python
Comments are closed.