GiulioPN's Blog

Python packaging

Python is a highly versatile programming language that can be used in various ways:

  1. Scripting, a mix of python modules
  2. Interactive mode, using tools like IPython or Jupyter Notebooks
  3. Building comprehensive programming systems with Python packages

In my personal experience, beginner developers may find these different usages confusing. I've encountered situations where a mix of these implementations was deployed, which can be particularly frustrating for an experienced developer.

In this article I would like to exploit the good practie to create a basic python package which can be easily installable, deployable or deliverable.

Python Packaging

A simple python package can be implemented as follow:

bpa/                    
├── bpa/
│   └── __init__.py
│   └── src/...
├── docs/...
├── scripts/...
├── tests/...
├── pyproject.toml
├── LICENSE.txt
└── README.md

In this case the package name is bpa and its components are:

  • bpa/: This is the main package directory. It contains the Python modules (in the src\ folder) and an __init__.py file to mark it as a package. In particular, the init file should contain the version of the package (e.g. __version__ = "1.0.0.dev0").
  • docs/: This directory contains the project documentation.
  • scripts/: This folder contains some additional scripts, like script of example
  • tests/: This directory contains test modules.
  • pyproject.toml: This is the configuration file for your project, specifying build system requirements and project metadata.
  • LICENSE.txt: This file contains the licence of the package.
  • README.md: This file provides an overview of the project documentation.

The pyproject.toml file is a configuration file used in Python projects to define build system requirements and project metadata. It was introduced as part of PEP 518 and is designed to standardize project configuration and support multiple build tools. In particular we have that:

  • The Build System Requirements specifies the build backend like Setuptools, Hatchling, Flit, or PDM and any dependencies needed to build the project. The build system generate the binary code, see more in the Build backend section.
  • The Project Metadata can contain metadata about the project such as name, version, authors, dependencies...
  • The Tool-Specific Configuration can include configuration sections for various tools (e.g., linters, formatters).

Although pyproject.toml is widely used in the modern python project, the legacy packeges can have different types of file for the specification and definition of the project metadata and build system. See more in section Legacy configuration files.

Build backend

A python build backend is a tool that takes the python package source code as input and can generete one of the following:

  • A sdist called source distribution. It is a versioned archive file (.tar.gz) which contains the package source code. Moreover, it contains an additional special file called PKG-INFO, which holds the project metadata core metadata specifications.
  • A bdist called build distribution.

So if the sdist can be view as a simple archive of the source code, the bdist is more complex. The standard build distribution is the Wheel format introduced in the PEP 427.

Important to note that for each fixed version of our python package exit only one sdist, but there may be many wheels.

A distribtion can build with different build backend, but not all are accepted by the official python standard. Following some popular Python build backend tools:

  • Setuptools: The most widely used and traditional build backend for Python projects.
  • Poetry: A modern tool that combines dependency management and packaging. Not included in the PEPs.
  • Flit: A simple and fast tool for packaging Python projects.
  • Hatchling: A newer tool focused on being fast and easy to use, part of the Hatch project.
  • PDM: A modern Python package manager and build tool, following PEP 582.
  • Meson: A build system that can be used with Python via the meson-python plugin.

The package distribution

To facilitate installation, deployment, and delivery, you need to build a distribution. First, install the build package:

python3 -m pip install build

To create a source distribution (sdist) on a Unix/MacOS system:

python3 -m build --sdist

To create a binary distribution (bdist, specifically a Wheel) on a Unix/MacOS system:

python3 -m build --wheel

After creating a distribution (or “package”) for your project, you may need to make it installable from the Python Package Index (PyPI). For detailed information on uploading a Python package, refer to the official guide on uploading your project to PyPI.

Working in debug mode

It is possible to install a project in “editable” or “develop” mode while you’re working on it.

To install a Python package in “editable”/”development” mode Change directory to the root of the project directory and run:

python3 -m pip install -e .

Notice that the pip command-line flag -e is short for --editable, and . refers to the current working directory, so together, it means to install the current directory (i.e. your project) in editable mode.

Legacy configuration files

The legacy configuration files could be:

  • setup.py: The traditional script for configuring the build and installation of Python packages
from setuptools import setup

setup(
   name="my_project",
   version="0.1.0",
   description="A short description",
   author="Your Name",
   packages=["my_project"],
   install_requires=["requests>=2.25.1"],
   )

   [metadata]
   name = my_project
   version = 0.1.0
   description = A short description

   [options]
   packages = find:
   install_requires =
   requests>=2.25.1
  • requirements.txt: The file contains the dependencies for the project, typically used with pip
requests==2.25.1
pytest==6.2.2
  • requirements.yml: Typically used for specifying dependencies in environments managed by tools like Conda. More information on conda packaging documentation
name: my_project_env
dependencies:
   - python=3.8
   - requests=2.25.1
   - pytest=6.2.2

From a development point of view, their differences can be summarized as follows:

  • Declarative vs. Scripted: pyproject.toml, setup.cfg, requirements.txt, and requirements.yml are declarative, while setup.py is scripted.
  • Standardization: pyproject.toml aims to standardize configurations across tools, while setup.py and setup.cfg are specific to setuptools.
  • Build System Specification: pyproject.toml includes build system requirements, which is not handled by setup.py, setup.cfg, or requirements.txt.
  • Tool-Specific Configuration: pyproject.toml can contain configuration for various tools, while setup.py and setup.cfg are primarily for packaging.
  • Dependency Management: requirements.txt and requirements.yml are focused solely on dependencies, often for deployment environments, whereas pyproject.toml can include both dependencies and metadata.

The pyproject.toml is the best choice since it offers a modern, standardized way to configure all the aspects of a Python project.

References

[1]: Latest Python packaging overview
[2]: Python packaging format official website
[3]: Stack overflow Python configuration files differences
[4]: Python working mode
[5]: Python packagin glossary

Published

Category

python

Tags

Contact