Python 101 – How to Create a Python Package

/ intermediate, Python / By / / 30DaysOfPython, Python

When you create a Python file, you are creating a Python module. Any Python file that you create can be imported by another Python script. Thus, by definition, it is also a Python module. If you have two or more related Python files, then you may have a Python package.

Some organizations keep all their code to themselves. This is known as closed-source. Python is an open-source language and most of the Python modules and packages that you can get from the Python Package Index (PyPI) are all free and open-source as well. One of the quickest ways to share your package or module is to upload it to the Python Package Index or Github or both.

In this article, you will learn about the following topics:

  • Creating a Module
  • Creating a Package
  • Packaging a Project for PyPI
  • Creating Project Files
  • Creating setup.py
  • Uploading to PyPI

The first step in the process is to understand what creating a reusable module looks like. Let’s get started!

Creating a Module

Any Python file you create is a module that you can import. You can try it out with some of the examples from this book by adding a new file to any of the articles’ code folders and attempt to import one of the modules in there. For example, if you have a Python file named a.py and then create a new file named b.py, you can import a into b through the use of import a.

Of course, that’s a silly example. Instead, you will create a simple module that has some basic arithmetic functions in it. You can name the file arithmetic.py and add this code to it:

# arithmetic.py

def add(x, y):
    return x + y

def divide(x, y):
    return x / y

def multiply(x, y):
    return x * y

def subtract(x, y):
    return x - y

This code is very naive. You have no error handling at all, for example. What that means is that you could divide by zero and cause an exception to be thrown. You could also pass incompatible types to these functions, like a string and an integer — that would cause a different kind of exception to be raised.

However, for learning purposes, this code is adequate. You can prove that it is importable by creating a test file. Create a new file named test_arithmetic.py and add this code to it:

# test_arithmetic.py

import arithmetic
import unittest

class TestArithmetic(unittest.TestCase):

    def test_addition(self):
        self.assertEqual(arithmetic.add(1, 2), 3)

    def test_subtraction(self):
        self.assertEqual(arithmetic.subtract(2, 1), 1)

    def test_multiplication(self):
        self.assertEqual(arithmetic.multiply(5, 5), 25)

    def test_division(self):
        self.assertEqual(arithmetic.divide(8, 2), 4)

if __name__ == '__main__':
    unittest.main()

Save your test in the same folder as your module. Now you can run this code using the following command:

$ python3 test_arithmetic.py 
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

This demonstrates that you can import arithmetic.py as a module. These tests also show the basic functionality of the code works. You can enhance these tests by testing division by zero and mixing strings and integers. Those kinds of tests will currently fail. Once you have a failing test, you can follow the Test Driven Development methodology to fix the issues.

Now let’s find out how to make a Python package!

Creating a Package

A Python package is one or more files that you plan on sharing with others, usually by uploading it to the Python Package Index (PyPI). Packages are generally made by naming a directory of files rather than a file itself. Then inside of that directory you will have a special __init__.py file. When Python sees the __init__.py file, it knows that the folder is importable as a package.

There are a couple ways to transform arithmetic.py into a package. The simplest is to move the code from arithmetic.py into arithmetic/__init__.py:

  • create the folder arithmetic
  • move/copy arithmetic.py to arithmetic/__init__.py
  • if you used “copy” in the previous step then delete arithmetic.py
  • run test_arithmetic.py

That last step is extremely important! If your tests still pass then you know your conversion from a module to a package worked. To test out your package, open up a Command Prompt if you’re on Windows or a terminal if you’re on Mac or Linux. Then navigate to the folder that contains the arithmetic folder, but not inside of it. You should now be in the same folder as your test_arithmetic.py file. At this point you can run python test_arithmetic.py and see if your efforts were successful.

It might seem silly to simply put all your code in a single __init__.py file, but that actually works fine for files up to a few thousand lines.

The second way to transform arithmetic.py into a package is similar to the first, but involves using more files than just __init__.py. In real code the functions/classes/etc. in each file would be grouped somehow — perhaps one file for all your package’s custom exceptions, one file for common utilities, and one file for the main functionality.

For our example, you’ll just split the four functions in arithmetic.py into their own files. Go ahead and move each function from __init__.py into its own file. Your folder structure should look like this:

arithmetic/
    __init__.py
    add.py
    subtract.py
    multiply.py
    divide.py

For the __init__.py file, you can add the following code:

# __init__.py
from .add import add
from .subtract import subtract
from .multiply import multiply
from .divide import divide

Now that you’ve made these changes what should be your next step? Hopefully, you said, “Run my tests!” If your tests still pass then you haven’t broken your API.

Right now your arithmetic package is only available to your other Python code if you happen to be in the same folder as your test_arithmetic.py file. To make it available in your Python session or in other Python code you can use Python’s sys module to add your package to the Python search path. The search path is what Python uses to find modules when you use the import keyword. You can see what paths Python searches by printing out sys.path.

Let’s pretend that your arithmetic folder is in this location: /Users/michael/packages/arithmetic. To add that to Python’s search path, you can do this:

import sys

sys.path.append("/Users/michael/packages/arithmetic")
import arithmetic

print(arithmetic.add(1, 2))

This will add arithmetic to Python’s path so you can import it and then use the package in your code. However, that’s really awkward. It would be nice if you could install your package using pip so you don’t have to mess around with the path all the time.

Let’s find out how to do that next!

Packaging a Project for PyPI

When it comes to creating a package for the Python Package Index (PyPI), you will need some additional files. There is a good tutorial on the process for creating and uploading a package to PyPI here:

The official packaging instructions recommend that you set up a directory structure like this:

my_package/
    LICENSE
    README.md
    arithmetic/
        __init__.py
        add.py
        subtract.py
        multiply.py
        divide.py
    setup.py
    tests/

The tests folder can be empty. This is the folder where you would include tests for your package. Most developers use Python’s unittest or the pytest framework for their tests. For this example, you can leave the folder empty.

Let’s move on and learn about the other files you need to create in the next section!

Creating Project Files

The LICENSE file is where you mention what license your package has. This tells the users of the package what they can and cannot do with your package. There are a lot of different licenses you can use. The GPL and MIT licenses are just a couple of popular examples.

The README.md file is a description of your project, written in Markdown. You will want to write about your project in this file and include any information about dependencies that it might need. You can give instructions for installation as well as example usage of your package. Markdown is quite versatile and even let’s you do syntax highlighting!

The other file you need to supply is setup.py. That file is more complex, so you’ll learn about that in the next section.

Creating setup.py

There is a special file named setup.py that is used as a build script for Python distributions. It is used by setuptools, which does the actual building for you. If you’d like to know more about setuptools, then you should check out the following:

You can use the setup.py to create a Python wheel. The wheel is a ZIP-format archive with a specially formatted name and a .whl extension. It contains everything necessary to install your package. You can think of it as a zipped version of your code that pip can unzip and install for you. The wheel follows PEP 376, which you can read about here:

Once you’re done reading all that documentation (if you wanted to), you can create your setup.py and add this code to it:

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="arithmetic-YOUR-USERNAME-HERE", # Replace with your own username
    version="0.0.1",
    author="Mike Driscoll",
    author_email="driscoll@example.com",
    description="A simple arithmetic package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/driscollis/arithmetic",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)

The first step in this code is to import setuptools. Then you read in your README.md file into a variable that you will use soon. The last bit is the bulk of the code. Here you call setuptools.setup(), which can take in quite a few different arguments. The example above is only a sampling of what you can pass to this function. To see the full listing, you’ll need to go here:

Most of the arguments are self-explanatory. Let’s focus on the more obtuse ones. The packages arguments is a list of packages needed for your package. In this case, you use find_packages() to find the necessary packages for you automatically. The classifiers argument is used for passing additional metadata to pip. For example, this code tells pip that the package is Python 3 compatible.

Now that you have a setup.py, you are ready to create a Python wheel!

Generating a Python Wheel

The setup.py is used to create Python wheels. It’s always a good idea to make sure you have the latest version of setuptools and wheel installed, so before you create your own wheel, you should run the following command:

python3 -m pip install --user --upgrade setuptools wheel

This will update the packages if there is a newer version than the one you currently have installed. Now you are ready to create a wheel yourself. Open up a Command Prompt or terminal application and navigate to the folder that contains your setup.py file. Then run the following command:

python3 setup.py sdist bdist_wheel

This command will output a lot of text, but once it has finished you will find a new folder named dist that contains the following two files:

  • arithmetic_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
  • arithmetic-YOUR-USERNAME-HERE-0.0.1.tar.gz

The tar.gz is a source archive, which means it has the Python source code for your package inside of it. Your users can use the source archive to build the package on their own machines, if they need to. The whl format is an archive that is used by pip to install your package on your user’s machine.

You can install the wheel using pip directly, if you want to:

python3 -m pip install arithmetic_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl

But the normal method would be to upload your package to the Python Package Index (PyPI) and then install it. Let’s discover how to get your amazing package on PyPI next!

Uploading to PyPI

The first step to upload a package to PyPI is to create an account on Test PyPI. This allows you to test that your package can be uploaded on a test server and installed from that test server. To create an account, go to the following URL and follow the steps on that page:

Now you need to create a PyPI API token. This will allow you to upload the package securely. To get the API token, you’ll need to go here:

You can limit a token’s scope. However, you don’t need to do that for this token as you are creating it for a new project. Make sure you copy the token and save it off somewhere BEFORE you close the page. Once the page is closed, you cannot retrieve the token again. You will be required to create a new token instead.

Now that you are registered and have an API token, you will need to get the twine package. You will use twine to upload your package to PyPI. To install twine, you can use pip like this:

python3 -m pip install --user --upgrade twine

Once installed, you can upload your package to Test PyPI using the following command:

python3 -m twine upload --repository testpypi dist/*

Note that you will need to run this command from within the folder that contains the setup.py file as it is copying all the files in the dist folder to Test PyPI. When you run this command, it will prompt you for a username and password. For the username, you need to use __token__. The password is the token value that is prefixed with pypi-.

When this command runs, you should see output similar to the following:

Uploading distributions to https://test.pypi.org/legacy/
Enter your username: [your username]
Enter your password:
Uploading arithmetic_YOUR_USERNAME_HERE-0.0.1-py3-none-any.whl
100%|?????????????????????| 4.65k/4.65k [00:01<00:00, 2.88kB/s]
Uploading arithmetic_YOUR_USERNAME_HERE-0.0.1.tar.gz
100%|?????????????????????| 4.25k/4.25k [00:01<00:00, 3.05kB/s]

At this point, you should now be able to view your package on Test PyPI at the following URL:

Now you can test installing your package from Test PyPI by using the following command:

python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps arithmetic-YOUR-USERNAME-HERE

If everything worked correctly, you should now have the arithmetic package installed on your system. Of course, this tutorial showed you how to package things up for Test PyPI. Once you have verified that it works, then you will need to do the following to install to the real PyPI:

  • Choose a memorable and unique name for the package
  • Register an account at https://pypi.org
  • Use twine upload dist/* to upload your package and enter your credentials for the account you registered on the real PyPI. You won’t need to use the --repository flag when uploading to the real PyPI as that server is the default
  • Install your package from the real PyPI by using pip install your_unique_package_name

Now you know how to distribute a package of your own creation on the Python Package Index!

Wrapping Up

Python modules and packages are what you import in your programs. They are, in many ways, the building blocks of your programs. In this article, you learned about the following:

  • Creating a Module
  • Creating a Package
  • Packaging a Project for PyPI
  • Creating Project Files
  • Creating setup.py
  • Uploading to PyPI

At this point, you not only know what modules and packages are, but also how to distribute them via the Python Package Index. Now you and any other Python developer can download and install your packages. Congratulations! You are now a package maintainer!

Related Reading

This article is based on a chapter from Python 101, 2nd Edition, which you can purchase on Leanpub or Amazon.

If you’d like to learn more Python, then check out these tutorials: