Build Python Packages: Development to PyPI Guide

Python packaging has evolved significantly, with modern tools and standards making it easier than ever to distribute your code.

This guide walks you through building professional Python packages and publishing them to PyPI. If you’re new to Python or need a quick reference, check out our Python Cheatsheet to get up to speed with Python fundamentals.

Why Package Your Python Code?

Packaging your Python project offers multiple benefits:

Reusability: Share code across multiple projects without copy-pasting
Distribution: Enable others to install your code with a simple pip install
Dependency Management: Clearly speci…

Python packaging has evolved significantly, with modern tools and standards making it easier than ever to distribute your code.

Why Package Your Python Code?

Packaging your Python project offers multiple benefits:

Reusability: Share code across multiple projects without copy-pasting
Distribution: Enable others to install your code with a simple pip install
Dependency Management: Clearly specify and manage dependencies
Versioning: Track releases and maintain backwards compatibility
Professional Standards: Follow Python community best practices
Documentation: Structure encourages proper documentation and testing

Modern Python Package Structure

A well-organized package follows this structure:

my-package/
├── src/
│   └── mypackage/
│       ├── __init__.py
│       ├── core.py
│       └── utils.py
├── tests/
│   ├── __init__.py
│   └── test_core.py
├── docs/
│   └── index.md
├── .github/
│   └── workflows/
│       └── publish.yml
├── pyproject.toml
├── README.md
├── LICENSE
└── .gitignore

The src-layout is now preferred over the flat layout because it:

Prevents accidental imports of uninstalled code during development
Makes testing more reliable by forcing installation
Clearly separates source code from other project files

When organizing your package’s internal structure, consider applying clean architecture principles to make your code more maintainable and testable. Our guide on Python Design Patterns for Clean Architecture covers SOLID principles, dependency injection, and layered architecture patterns that work excellently with Python packages.

The pyproject.toml File: Modern Configuration

pyproject.toml (PEP 518, 621) is the modern standard for Python project configuration, replacing the legacy setup.py:

[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.1.0"
description = "A fantastic Python package"
readme = "README.md"
requires-python = ">=3.8"
license = {text = "MIT"}
authors = [
{name = "Your Name", email = "you@example.com"}
]
keywords = ["example", "tutorial", "package"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
dependencies = [
"requests>=2.28.0",
"click>=8.1.0",
]

[project.optional-dependencies]
dev = [
"pytest>=7.4.0",
"black>=23.0.0",
"flake8>=6.0.0",
"mypy>=1.5.0",
]
docs = [
"sphinx>=7.0.0",
"sphinx-rtd-theme>=1.3.0",
]

[project.urls]
Homepage = "https://github.com/yourusername/mypackage"
Documentation = "https://mypackage.readthedocs.io"
Repository = "https://github.com/yourusername/mypackage"
Issues = "https://github.com/yourusername/mypackage/issues"

[project.scripts]
mypackage-cli = "mypackage.cli:main"

[tool.setuptools.packages.find]
where = ["src"]

Key Configuration Sections

build-system: Specifies build backend (setuptools, hatchling, poetry-core, flit-core)
project: Core metadata and dependencies
project.optional-dependencies: Extra feature dependencies (dev, docs, testing)
project.scripts: Command-line entry points
tool.*: Tool-specific configuration (black, pytest, mypy, etc.)

Choosing a Build Backend

Setuptools (Standard Choice)

The most widely used and compatible option:

[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.build_meta"

Pros: Universal compatibility, extensive features, large ecosystem Cons: More verbose configuration, slower than newer alternatives

Hatchling (Modern & Fast)

Lightweight and performant modern backend:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Pros: Fast builds, simple config, good defaults Cons: Fewer plugins than setuptools

Poetry (All-in-One)

Complete dependency and environment management:

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = "A fantastic Python package"
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.8"
requests = "^2.28.0"

Pros: Dependency resolution, lock files, integrated workflow Cons: Different workflow, larger tool surface area

Flit (Minimalist)

Simple tool for simple packages:

[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"

Pros: Extremely simple, minimal configuration Cons: Limited features for complex packages

Building Your Package

Install Build Tools

# Install the modern build tool
pip install build

# Or for Poetry
pip install poetry

# Or for Hatchling
pip install hatch

Build Distribution Files

Using the build package (works with any backend):

# Clean previous builds
rm -rf dist/ build/ *.egg-info

# Build source distribution and wheel
python -m build

# This creates:
# dist/mypackage-0.1.0.tar.gz    (source distribution)
# dist/mypackage-0.1.0-py3-none-any.whl  (wheel)

Using Poetry:

poetry build

Using Hatch:

hatch build

Understanding Distribution Formats

Source Distribution (sdist) - .tar.gz

Contains source code and build instructions
Users’ pip builds it during installation
Includes tests, docs, and other development files

Wheel - .whl

Pre-built binary distribution
Fast installation (no build step)
Platform-specific or pure Python
Recommended format for distribution

Publishing to PyPI

Method 1: Manual Upload with Twine (Traditional)

# Install twine
pip install twine

# Check package before uploading
twine check dist/*

# Upload to TestPyPI first (recommended)
twine upload --repository testpypi dist/*

# Test installation from TestPyPI
pip install --index-url https://test.pypi.org/simple/ mypackage

# Upload to production PyPI
twine upload dist/*

Configure credentials in ~/.pypirc:

[distutils]
index-servers =
pypi
testpypi

[pypi]
username = __token__
password = pypi-AgEIcHlwaS5vcmc...

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-AgENdGVzdC5weXBpLm9yZw...

Method 2: Trusted Publishing (Recommended for CI/CD)

Trusted Publishing uses OpenID Connect (OIDC) to authenticate from CI/CD platforms without storing tokens.

Setup in PyPI:

Go to your PyPI project settings
Navigate to “Publishing” section
Add a new “pending publisher”
Configure:

Owner: Your GitHub username/organization
Repository: Repository name
Workflow: publish.yml
Environment: release (optional but recommended)

GitHub Actions Workflow (.github/workflows/publish.yml):

name: Publish to PyPI

on:
release:
types: [published]

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'

- name: Install build dependencies
run: |
python -m pip install --upgrade pip
pip install build

- name: Build package
run: python -m build

- name: Store distribution packages
uses: actions/upload-artifact@v3
with:
name: python-package-distributions
path: dist/

publish:
needs: build
runs-on: ubuntu-latest
environment: release
permissions:
id-token: write

steps:
- name: Download distributions
uses: actions/download-artifact@v3
with:
name: python-package-distributions
path: dist/

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1

Advantages:

No API tokens to manage or secure
Automatic authentication via OIDC
Enhanced security through environment protection rules
Audit trail of all publications

Best Practices

1. Version Management

Use semantic versioning (SemVer): MAJOR.MINOR.PATCH

# Install version bumping tool
pip install bump2version

# Bump version
bump2version patch  # 0.1.0 -> 0.1.1
bump2version minor  # 0.1.1 -> 0.2.0
bump2version major  # 0.2.0 -> 1.0.0

Configure in .bumpversion.cfg:

[bumpversion]
current_version = 0.1.0
commit = True
tag = True

[bumpversion:file:pyproject.toml]
search = version = "{current_version}"
replace = version = "{new_version}"

[bumpversion:file:src/mypackage/__init__.py]
search = __version__ = "{current_version}"
replace = __version__ = "{new_version}"

2. Include Essential Files

README.md: Clear project description, installation, usage examples LICENSE: Choose appropriate license (MIT, Apache 2.0, GPL, etc.) CHANGELOG.md: Document changes between versions .gitignore: Exclude build artifacts, caches, virtual environments

3. Comprehensive Testing

# Install testing dependencies
pip install pytest pytest-cov

# Run tests with coverage
pytest --cov=mypackage tests/

# Generate coverage report
pytest --cov=mypackage --cov-report=html tests/

Configure in pyproject.toml:

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_functions = ["test_*"]
addopts = "--strict-markers --cov=mypackage"

4. Code Quality Tools

# Formatting
black src/ tests/

# Linting
flake8 src/ tests/

# Type checking
mypy src/

# Import sorting
isort src/ tests/

Integrate into pre-commit hooks (.pre-commit-config.yaml):

repos:
- repo: https://github.com/psf/black
rev: 23.9.1
hooks:
- id: black

- repo: https://github.com/pycqa/flake8
rev: 6.1.0
hooks:
- id: flake8

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.5.1
hooks:
- id: mypy

5. Documentation

Use Sphinx for documentation:

# Install Sphinx
pip install sphinx sphinx-rtd-theme

# Initialize docs
cd docs
sphinx-quickstart

# Build documentation
make html

Or use MkDocs for simpler Markdown-based docs:

pip install mkdocs mkdocs-material
mkdocs new .
mkdocs serve

6. Continuous Integration

Testing your Python packages across different platforms and environments is crucial for reliability. For insights on Python performance in different deployment scenarios, see our comparison of AWS Lambda performance: JavaScript vs Python vs Golang, which explores how Python performs in serverless environments.

Complete CI/CD workflow (.github/workflows/ci.yml):

name: CI

on:
push:
branches: [main, develop]
pull_request:
branches: [main]

jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']

steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev]"

- name: Run tests
run: pytest --cov=mypackage --cov-report=xml

- name: Upload coverage
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml

lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'

- name: Install dependencies
run: |
pip install black flake8 mypy

- name: Black formatting check
run: black --check src/ tests/

- name: Flake8
run: flake8 src/ tests/

- name: MyPy
run: mypy src/

Common Pitfalls and Solutions

Issue 1: Import Errors After Installation

Problem: Package installed but imports fail

Solution: Ensure proper package structure with __init__.py files and correct pyproject.toml configuration:

[tool.setuptools.packages.find]
where = ["src"]
include = ["mypackage*"]

Issue 2: Missing Dependencies

Problem: Package installs but fails at runtime due to missing dependencies

Solution: Declare all runtime dependencies in pyproject.toml:

[project]
dependencies = [
"requests>=2.28.0",
"click>=8.1.0",
]

Issue 3: Version Conflicts

Problem: Package works in development but fails in production

Solution: Use virtual environments and specify minimum versions:

# Create isolated environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install in editable mode for development
pip install -e ".[dev]"

Issue 4: Large Package Size

Problem: Package takes too long to download/install

Solution: Exclude unnecessary files using MANIFEST.in:

include README.md
include LICENSE
include pyproject.toml
recursive-include src *.py
recursive-exclude * __pycache__
recursive-exclude * *.py[co]
recursive-exclude tests *
recursive-exclude docs *

Issue 5: Platform-Specific Issues

Problem: Package works on one OS but fails on another

Solution: Test on multiple platforms using CI/CD matrix builds (see CI example above)

Publishing Checklist

Before publishing your package to PyPI, verify:

[ ] All tests pass across Python versions
[ ] Code is formatted and linted
[ ] README.md is complete with examples
[ ] LICENSE file is included
[ ] Version number is updated
[ ] CHANGELOG.md is updated
[ ] Dependencies are correctly specified
[ ] Package builds without errors (python -m build)
[ ] Package tested on TestPyPI
[ ] Documentation is up to date
[ ] Git repository is tagged with version
[ ] GitHub release is created (for trusted publishing)

Advanced Topics

Entry Points and CLI Tools

Create command-line interfaces:

[project.scripts]
mypackage = "mypackage.cli:main"

Implementation in src/mypackage/cli.py:

import click

@click.command()
@click.option('--name', default='World', help='Name to greet')
def main(name):
"""Simple CLI tool example"""
click.echo(f'Hello, {name}!')

if __name__ == '__main__':
main()

For a real-world example of creating Python packages that interact with external APIs, check out our guide on Integrating Ollama with Python, which demonstrates building Python clients with both REST API and official library integrations.

Plugin Systems

Enable plugin discovery:

[project.entry-points."mypackage.plugins"]
plugin_a = "mypackage_plugin_a:PluginA"

C Extensions

For performance-critical code:

[tool.setuptools.ext-modules]
name = "mypackage._speedups"
sources = ["src/mypackage/_speedups.c"]

Namespace Packages

For distributed packages under a common namespace:

[tool.setuptools.packages.find]
where = ["src"]
include = ["company.*"]

[tool.setuptools.package-data]
"company.mypackage" = ["py.typed"]

Useful Tools and Resources

Package Development Tools

cookiecutter: Project templates for Python packages
tox: Test automation across Python versions
nox: Flexible test automation (tox alternative)
pre-commit: Git hook framework for code quality
commitizen: Standardize commit messages and versioning

Documentation Platforms

Read the Docs: Free documentation hosting
GitHub Pages: Host MkDocs or Sphinx documentation
pdoc: Simple API documentation generator

Package Registries

PyPI: The official Python Package Index (pypi.org)
TestPyPI: Testing environment (test.pypi.org)
Anaconda.org: Conda package distribution
GitHub Packages: Private package hosting

Monitoring and Analytics

PyPI Stats: Download statistics for packages
Libraries.io: Dependency monitoring and alerts
Snyk: Security vulnerability scanning
Dependabot: Automated dependency updates

Conclusion

Modern Python packaging has evolved to be developer-friendly with standards like pyproject.toml, powerful build backends, and secure publishing via Trusted Publishing. Following this guide, you can create professional, maintainable Python packages that serve your users effectively.

Key takeaways:

Use pyproject.toml for all new projects
Choose the right build backend for your needs
Implement automated testing and CI/CD
Use Trusted Publishing for secure, token-free deployment
Follow semantic versioning and maintain changelogs
Provide comprehensive documentation and examples

The Python packaging ecosystem continues to improve with better tooling, standards, and security features. Stay updated with Python Enhancement Proposals (PEPs) and community best practices to keep your packages modern and maintainable.

Why Package Your Python Code?

Why Package Your Python Code?

Modern Python Package Structure

The pyproject.toml File: Modern Configuration

Key Configuration Sections

Choosing a Build Backend

Setuptools (Standard Choice)

Hatchling (Modern & Fast)

Poetry (All-in-One)

Flit (Minimalist)

Building Your Package

Install Build Tools

Build Distribution Files

Understanding Distribution Formats

Publishing to PyPI

Method 1: Manual Upload with Twine (Traditional)

Method 2: Trusted Publishing (Recommended for CI/CD)

Best Practices

1. Version Management

2. Include Essential Files

3. Comprehensive Testing

4. Code Quality Tools

5. Documentation

6. Continuous Integration

Common Pitfalls and Solutions

Issue 1: Import Errors After Installation

Issue 2: Missing Dependencies

Issue 3: Version Conflicts

Issue 4: Large Package Size

Issue 5: Platform-Specific Issues

Publishing Checklist

Advanced Topics

Entry Points and CLI Tools

Plugin Systems

C Extensions

Namespace Packages

Useful Tools and Resources

Package Development Tools

Documentation Platforms

Package Registries

Monitoring and Analytics

Conclusion

Useful Links

Other Useful Articles on this Site

Similar Posts