When you create a Python file, you are creating a Python module. Any Python file that you create can be imported by another Python script. Thus, by definition, it is also a Python module. If you have two or more related Python files, then you may have a Python package.
Some organizations keep all their code to themselves. This is known as closed-source. Python is an open-source language and most of the Python modules and packages that you can get from the Python Package Index (PyPI) are all free and open-source as well. One of the quickest ways to share your package or module is to upload it to the Python Package Index or Github or both.
In this article, you will learn about the following topics:
- Creating a Module
- Creating a Package
- Packaging a Project for PyPI
- Creating Project Files
- Uploading to PyPI
The first step in the process is to understand what creating a reusable module looks like. Let’s get started!
Creating a Module
Any Python file you create is a module that you can import. You can try it out with some of the examples from this book by adding a new file to any of the articles’ code folders and attempt to import one of the modules in there. For example, if you have a Python file named
a.py and then create a new file named
b.py, you can import
b through the use of
Of course, that’s a silly example. Instead, you will create a simple module that has some basic arithmetic functions in it. You can name the file
arithmetic.py and add this code to it:
# arithmetic.py def add(x, y): return x + y def divide(x, y): return x / y def multiply(x, y): return x * y def subtract(x, y): return x - y
This code is very naive. You have no error handling at all, for example. What that means is that you could divide by zero and cause an exception to be thrown. You could also pass incompatible types to these functions, like a string and an integer — that would cause a different kind of exception to be raised.
However, for learning purposes, this code is adequate. You can prove that it is importable by creating a test file. Create a new file named
test_arithmetic.py and add this code to it:
# test_arithmetic.py import arithmetic import unittest class TestArithmetic(unittest.TestCase): def test_addition(self): self.assertEqual(arithmetic.add(1, 2), 3) def test_subtraction(self): self.assertEqual(arithmetic.subtract(2, 1), 1) def test_multiplication(self): self.assertEqual(arithmetic.multiply(5, 5), 25) def test_division(self): self.assertEqual(arithmetic.divide(8, 2), 4) if __name__ == '__main__': unittest.main()
Save your test in the same folder as your module. Now you can run this code using the following command:
$ python3 test_arithmetic.py .... ---------------------------------------------------------------------- Ran 4 tests in 0.000s OK
This demonstrates that you can import
arithmetic.py as a module. These tests also show the basic functionality of the code works. You can enhance these tests by testing division by zero and mixing strings and integers. Those kinds of tests will currently fail. Once you have a failing test, you can follow the Test Driven Development methodology to fix the issues.
Now let’s find out how to make a Python package!
Creating a Package
A Python package is one or more files that you plan on sharing with others, usually by uploading it to the Python Package Index (PyPI). Packages are generally made by naming a directory of files rather than a file itself. Then inside of that directory you will have a special
__init__.py file. When Python sees the
__init__.py file, it knows that the folder is importable as a package.
There are a couple ways to transform
arithmetic.py into a package. The simplest is to move the code from
- create the folder
- if you used “copy” in the previous step then delete
That last step is extremely important! If your tests still pass then you know your conversion from a module to a package worked. To test out your package, open up a Command Prompt if you’re on Windows or a terminal if you’re on Mac or Linux. Then navigate to the folder that contains the
arithmetic folder, but not inside of it. You should now be in the same folder as your
test_arithmetic.py file. At this point you can run
python test_arithmetic.py and see if your efforts were successful.
It might seem silly to simply put all your code in a single
__init__.py file, but that actually works fine for files up to a few thousand lines.
The second way to transform
arithmetic.py into a package is similar to the first, but involves using more files than just
__init__.py. In real code the functions/classes/etc. in each file would be grouped somehow — perhaps one file for all your package’s custom exceptions, one file for common utilities, and one file for the main functionality.
For our example, you’ll just split the four functions in
arithmetic.py into their own files. Go ahead and move each function from
__init__.py into its own file. Your folder structure should look like this:
arithmetic/ __init__.py add.py subtract.py multiply.py divide.py
__init__.py file, you can add the following code:
# __init__.py from .add import add from .subtract import subtract from .multiply import multiply from .divide import divide
Now that you’ve made these changes what should be your next step? Hopefully, you said, “Run my tests!” If your tests still pass then you haven’t broken your API.
Right now your
arithmetic package is only available to your other Python code if you happen to be in the same folder as your
test_arithmetic.py file. To make it available in your Python session or in other Python code you can use Python’s
sys module to add your package to the Python search path. The search path is what Python uses to find modules when you use the
import keyword. You can see what paths Python searches by printing out
Let’s pretend that your
arithmetic folder is in this location:
/Users/michael/packages/arithmetic. To add that to Python’s search path, you can do this:
import sys sys.path.append("/Users/michael/packages/arithmetic") import arithmetic print(arithmetic.add(1, 2))
This will add
arithmetic to Python’s path so you can import it and then use the package in your code. However, that’s really awkward. It would be nice if you could install your package using
pip so you don’t have to mess around with the path all the time.
Let’s find out how to do that next!
Packaging a Project for PyPI
When it comes to creating a package for the Python Package Index (PyPI), you will need some additional files. There is a good tutorial on the process for creating and uploading a package to PyPI here:
The official packaging instructions recommend that you set up a directory structure like this:
my_package/ LICENSE README.md arithmetic/ __init__.py add.py subtract.py multiply.py divide.py setup.py tests/
tests folder can be empty. This is the folder where you would include tests for your package. Most developers use Python’s
unittest or the
pytest framework for their tests. For this example, you can leave the folder empty.
Let’s move on and learn about the other files you need to create in the next section!
Creating Project Files
The LICENSE file is where you mention what license your package has. This tells the users of the package what they can and cannot do with your package. There are a lot of different licenses you can use. The GPL and MIT licenses are just a couple of popular examples.
The README.md file is a description of your project, written in Markdown. You will want to write about your project in this file and include any information about dependencies that it might need. You can give instructions for installation as well as example usage of your package. Markdown is quite versatile and even let’s you do syntax highlighting!
The other file you need to supply is
setup.py. That file is more complex, so you’ll learn about that in the next section.
There is a special file named
setup.py that is used as a build script for Python distributions. It is used by
setuptools, which does the actual building for you. If you’d like to know more about
setuptools, then you should check out the following:
You can use the
setup.py to create a Python wheel. The wheel is a ZIP-format archive with a specially formatted name and a
.whl extension. It contains everything necessary to install your package. You can think of it as a zipped version of your code that
pip can unzip and install for you. The wheel follows PEP 376, which you can read about here:
Once you’re done reading all that documentation (if you wanted to), you can create your
setup.py and add this code to it:
import setuptools with open("README.md", "r") as fh: long_description = fh.read() setuptools.setup( name="arithmetic-YOUR-USERNAME-HERE", # Replace with your own username version="0.0.1", author="Mike Driscoll", author_email="email@example.com", description="A simple arithmetic package", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/driscollis/arithmetic", packages=setuptools.find_packages(), classifiers=[ "Programming Language :: Python :: 3", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", ], python_requires='>=3.6', )
The first step in this code is to import
setuptools. Then you read in your
README.md file into a variable that you will use soon. The last bit is the bulk of the code. Here you call
setuptools.setup(), which can take in quite a few different arguments. The example above is only a sampling of what you can pass to this function. To see the full listing, you’ll need to go here:
Most of the arguments are self-explanatory. Let’s focus on the more obtuse ones. The
packages arguments is a list of packages needed for your package. In this case, you use
find_packages() to find the necessary packages for you automatically. The
classifiers argument is used for passing additional metadata to
pip. For example, this code tells
pip that the package is Python 3 compatible.
Now that you have a
setup.py, you are ready to create a Python wheel!
Generating a Python Wheel
setup.py is used to create Python wheels. It’s always a good idea to make sure you have the latest version of
wheel installed, so before you create your own wheel, you should run the following command:
python3 -m pip install --user --upgrade setuptools wheel
This will update the packages if there is a newer version than the one you currently have installed. Now you are ready to create a wheel yourself. Open up a Command Prompt or terminal application and navigate to the folder that contains your
setup.py file. Then run the following command:
python3 setup.py sdist bdist_wheel
This command will output a lot of text, but once it has finished you will find a new folder named
dist that contains the following two files:
tar.gz is a source archive, which means it has the Python source code for your package inside of it. Your users can use the source archive to build the package on their own machines, if they need to. The
whl format is an archive that is used by
pip to install your package on your user’s machine.
You can install the wheel using
pip directly, if you want to:
python3 -m pip install arithmetic_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
But the normal method would be to upload your package to the Python Package Index (PyPI) and then install it. Let’s discover how to get your amazing package on PyPI next!
Uploading to PyPI
The first step to upload a package to PyPI is to create an account on Test PyPI. This allows you to test that your package can be uploaded on a test server and installed from that test server. To create an account, go to the following URL and follow the steps on that page:
Now you need to create a PyPI API token. This will allow you to upload the package securely. To get the API token, you’ll need to go here:
You can limit a token’s scope. However, you don’t need to do that for this token as you are creating it for a new project. Make sure you copy the token and save it off somewhere BEFORE you close the page. Once the page is closed, you cannot retrieve the token again. You will be required to create a new token instead.
Now that you are registered and have an API token, you will need to get the
twine package. You will use
twine to upload your package to PyPI. To install
twine, you can use
pip like this:
python3 -m pip install --user --upgrade twine
Once installed, you can upload your package to Test PyPI using the following command:
python3 -m twine upload --repository testpypi dist/*
Note that you will need to run this command from within the folder that contains the
setup.py file as it is copying all the files in the
dist folder to Test PyPI. When you run this command, it will prompt you for a username and password. For the username, you need to use
__token__. The password is the token value that is prefixed with
When this command runs, you should see output similar to the following:
Uploading distributions to https://test.pypi.org/legacy/ Enter your username: [your username] Enter your password: Uploading arithmetic_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl 100%|?????????????????????| 4.65k/4.65k [00:01<00:00, 2.88kB/s] Uploading arithmetic_YOUR_USERNAME_HERE-0.0.1.tar.gz 100%|?????????????????????| 4.25k/4.25k [00:01<00:00, 3.05kB/s]
At this point, you should now be able to view your package on Test PyPI at the following URL:
Now you can test installing your package from Test PyPI by using the following command:
python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps arithmetic-YOUR-USERNAME-HERE
If everything worked correctly, you should now have the
arithmetic package installed on your system. Of course, this tutorial showed you how to package things up for Test PyPI. Once you have verified that it works, then you will need to do the following to install to the real PyPI:
- Choose a memorable and unique name for the package
- Register an account at https://pypi.org
twine upload dist/*to upload your package and enter your credentials for the account you registered on the real PyPI. You won’t need to use the
--repositoryflag when uploading to the real PyPI as that server is the default
- Install your package from the real PyPI by using
pip install your_unique_package_name
Now you know how to distribute a package of your own creation on the Python Package Index!
Python modules and packages are what you import in your programs. They are, in many ways, the building blocks of your programs. In this article, you learned about the following:
- Creating a Module
- Creating a Package
- Packaging a Project for PyPI
- Creating Project Files
- Uploading to PyPI
At this point, you not only know what modules and packages are, but also how to distribute them via the Python Package Index. Now you and any other Python developer can download and install your packages. Congratulations! You are now a package maintainer!
If you’d like to learn more Python, then check out these tutorials: