Create and Release a Private Python Package on GitHub

Introduction

As software engineers, we often find ourselves reusing code across different modules and projects. But let’s face it, this repetition creates a challenge: when we need to adjust or fix that code, we have to make the same changes in multiple places. For those of us who value efficiency and automation, the solution is clear – create a separate package that can be installed and used across our projects.
However, when dealing with confidential code, we can’t simply publish our package on public repositories like PyPI. Instead, we need to deploy it to a private repository such as GitHub or GitLab. This approach allows us to maintain security while still benefiting from the convenience of a reusable package.

In this tutorial, we’ll guide you through the process of:

Creating a Python package
Deploying the package to a private repository (GitHub)
Installing the package in a virtual environment (venv)

By following these steps, you’ll be able to reduce code duplication and simplify maintenance of shared code across your projects.

Note: DRY doesn’t just stand for “Don’t Repeat Yourself” – it’s also a lifestyle choice.

1. Setting Up Your Project Structure

First, let’s set up a basic project structure for our Python package:


<span>my</span><span>-</span><span>package</span><span>/</span>
<span>├──</span> <span>my_package</span><span>/</span>
<span>│</span>   <span>├──</span> <span>__init__</span><span>.</span><span>py</span>
<span>│</span>   <span>└──</span> <span>module1</span><span>.</span><span>py</span>
<span>├──</span> <span>setup</span><span>.</span><span>py</span>
<span>├──</span> <span>build</span><span>.</span><span>pipeline</span><span>.</span><span>yml</span>
<span>├──</span> <span>requirements</span><span>.</span><span>txt</span>
<span>├──</span> <span>.</span><span>gitignore</span>
<span>├──</span> <span>README</span><span>.</span><span>md</span>
<span>├──</span> <span>MANIFEST</span><span>.</span><span>in</span>
<span>└──</span> <span>LICENSE</span>
<span>my</span><span>-</span><span>package</span><span>/</span>
<span>├──</span> <span>my_package</span><span>/</span>
<span>│</span>   <span>├──</span> <span>__init__</span><span>.</span><span>py</span>
<span>│</span>   <span>└──</span> <span>module1</span><span>.</span><span>py</span>
<span>├──</span> <span>setup</span><span>.</span><span>py</span>
<span>├──</span> <span>build</span><span>.</span><span>pipeline</span><span>.</span><span>yml</span>
<span>├──</span> <span>requirements</span><span>.</span><span>txt</span>
<span>├──</span> <span>.</span><span>gitignore</span>
<span>├──</span> <span>README</span><span>.</span><span>md</span>
<span>├──</span> <span>MANIFEST</span><span>.</span><span>in</span>
<span>└──</span> <span>LICENSE</span>
my-package/
├── my_package/
│   ├── __init__.py
│   └── module1.py
├── setup.py
├── build.pipeline.yml
├── requirements.txt
├── .gitignore
├── README.md
├── MANIFEST.in
└── LICENSE

Enter fullscreen mode Exit fullscreen mode

Let’s break down the anatomy of our private Python package. Each file and directory plays a crucial role in making our package functional and installable:

my-package/: This is the root directory of our project. It’s like a house that contains all the rooms (files) we need.
my_package/: This subdirectory is where the actual Python code lives. It’s named the same as our package for clarity.
- __init__.py: This file makes Python treat the directory as a package. It can be empty or can execute the initialization code for the package.
- module1.py: This is where we put our main code. You can have multiple module files depending on your package’s complexity.
setup.py: Think of this as the instruction manual for our package. It contains metadata about our package (like its name and version) and lists its dependencies. This file is essential for making our package installable via pip.
requirements.txt: This file lists all the external Python packages our project depends on. It’s like a shopping list for pip, telling it exactly what to install to make our package work.
README.md: This is the welcome mat of our project. It’s usually the first thing people see when they visit our GitHub repository, so we use it to explain what our package does, how to install it, and how to use it.
.gitignore: This file tells Git which files or directories to ignore. It’s handy for keeping compiled code, temporary files, or sensitive information out of version control.
LICENSE: This file specifies how others can use, modify, or distribute our package. It’s crucial for open-source projects and helps protect our work.
MANIFEST.in: This file is used to include non-Python files in our package distribution. If we have data files, documentation, or other resources that need to be included, we list them here.
build.pipeline.yml: This file defines our Continuous Integration/Continuous Deployment (CI/CD) pipeline. It automates tasks like running tests and building the package when we push changes to our GitHub repository.

2. Creating the Package Code

Let’s create a simple module within our package. In my_package/module1.py:


<span>class</span> <span>Hello</span><span>:</span>
    <span>def</span> <span>__init__</span><span>(</span><span>self</span><span>,</span> <span>name</span><span>):</span>
        <span>self</span><span>.</span><span>name</span> <span>=</span> <span>name</span>
    <span>def</span> <span>greet</span><span>(</span><span>self</span><span>):</span>
        <span>return</span> <span>f</span><span>"</span><span>Hello, </span><span>{</span><span>self</span><span>.</span><span>name</span><span>}</span><span>!</span><span>"</span>
<span>class</span> <span>Hello</span><span>:</span>
    <span>def</span> <span>__init__</span><span>(</span><span>self</span><span>,</span> <span>name</span><span>):</span>
        <span>self</span><span>.</span><span>name</span> <span>=</span> <span>name</span>

    <span>def</span> <span>greet</span><span>(</span><span>self</span><span>):</span>
        <span>return</span> <span>f</span><span>"</span><span>Hello, </span><span>{</span><span>self</span><span>.</span><span>name</span><span>}</span><span>!</span><span>"</span>
class Hello:
    def __init__(self, name):
        self.name = name

    def greet(self):
        return f"Hello, {self.name}!"

Enter fullscreen mode Exit fullscreen mode

In my_package/__init__.py, we’ll import our module:


<span>from</span> <span>.module1</span> <span>import</span> <span>Hello</span>
<span>from</span> <span>.module1</span> <span>import</span> <span>Hello</span>
from .module1 import Hello

Enter fullscreen mode Exit fullscreen mode

3. Creating `setup.py`

The setup.py file is crucial for packaging our project. Here’s a basic example:


<span>from</span> <span>setuptools</span> <span>import</span> <span>setup</span><span>,</span> <span>find_packages</span>
<span>with</span> <span>open</span><span>(</span><span>'</span><span>requirements.txt</span><span>'</span><span>)</span> <span>as</span> <span>f</span><span>:</span>
    <span>requirements</span> <span>=</span> <span>f</span><span>.</span><span>read</span><span>().</span><span>splitlines</span><span>()</span>
<span>setup</span><span>(</span>
    <span>name</span><span>=</span><span>"</span><span>my_package</span><span>"</span><span>,</span>
    <span>version</span><span>=</span><span>"</span><span>0.1</span><span>"</span><span>,</span>
    <span>include_package_data</span><span>=</span><span>True</span><span>,</span>
    <span>python_requires</span><span>=</span><span>'</span><span>>=3.8</span><span>'</span><span>,</span>
    <span>packages</span><span>=</span><span>find_packages</span><span>(),</span>
    <span>setup_requires</span><span>=</span><span>[</span><span>'</span><span>setuptools-git-versioning</span><span>'</span><span>],</span>
    <span>install_requires</span><span>=</span><span>requirements</span><span>,</span>
    <span>author</span><span>=</span><span>"</span><span>Abdellah HALLOU</span><span>"</span><span>,</span>
    <span>author_email</span><span>=</span><span>"</span><span>abdeallahhallou33@gmail.com</span><span>"</span><span>,</span>
    <span>description</span><span>=</span><span>"</span><span>A short description of your package</span><span>"</span><span>,</span>
    <span>long_description</span><span>=</span><span>open</span><span>(</span><span>'</span><span>README.md</span><span>'</span><span>).</span><span>read</span><span>(),</span>
    <span>long_description_content_type</span><span>=</span><span>"</span><span>text/markdown</span><span>"</span><span>,</span>
    <span>classifiers</span><span>=</span><span>[</span>
        <span>"</span><span>Programming Language :: Python :: 3.8</span><span>"</span><span>,</span>
        <span>"</span><span>License :: OSI Approved :: MIT License</span><span>"</span><span>,</span>
        <span>"</span><span>Operating System :: OS Independent</span><span>"</span><span>,</span>
    <span>],</span>
    <span>version_config</span><span>=</span><span>{</span>
       <span>"</span><span>dirty_template</span><span>"</span><span>:</span> <span>"</span><span>{tag}</span><span>"</span><span>,</span>
    <span>}</span>
<span>)</span>
<span>from</span> <span>setuptools</span> <span>import</span> <span>setup</span><span>,</span> <span>find_packages</span>

<span>with</span> <span>open</span><span>(</span><span>'</span><span>requirements.txt</span><span>'</span><span>)</span> <span>as</span> <span>f</span><span>:</span>
    <span>requirements</span> <span>=</span> <span>f</span><span>.</span><span>read</span><span>().</span><span>splitlines</span><span>()</span>

<span>setup</span><span>(</span>
    <span>name</span><span>=</span><span>"</span><span>my_package</span><span>"</span><span>,</span>
    <span>version</span><span>=</span><span>"</span><span>0.1</span><span>"</span><span>,</span>
    <span>include_package_data</span><span>=</span><span>True</span><span>,</span>
    <span>python_requires</span><span>=</span><span>'</span><span>>=3.8</span><span>'</span><span>,</span>
    <span>packages</span><span>=</span><span>find_packages</span><span>(),</span>
    <span>setup_requires</span><span>=</span><span>[</span><span>'</span><span>setuptools-git-versioning</span><span>'</span><span>],</span>
    <span>install_requires</span><span>=</span><span>requirements</span><span>,</span>
    <span>author</span><span>=</span><span>"</span><span>Abdellah HALLOU</span><span>"</span><span>,</span>
    <span>author_email</span><span>=</span><span>"</span><span>abdeallahhallou33@gmail.com</span><span>"</span><span>,</span>
    <span>description</span><span>=</span><span>"</span><span>A short description of your package</span><span>"</span><span>,</span>
    <span>long_description</span><span>=</span><span>open</span><span>(</span><span>'</span><span>README.md</span><span>'</span><span>).</span><span>read</span><span>(),</span>
    <span>long_description_content_type</span><span>=</span><span>"</span><span>text/markdown</span><span>"</span><span>,</span>
    <span>classifiers</span><span>=</span><span>[</span>
        <span>"</span><span>Programming Language :: Python :: 3.8</span><span>"</span><span>,</span>
        <span>"</span><span>License :: OSI Approved :: MIT License</span><span>"</span><span>,</span>
        <span>"</span><span>Operating System :: OS Independent</span><span>"</span><span>,</span>
    <span>],</span>
    <span>version_config</span><span>=</span><span>{</span>
       <span>"</span><span>dirty_template</span><span>"</span><span>:</span> <span>"</span><span>{tag}</span><span>"</span><span>,</span>
    <span>}</span>
<span>)</span>
from setuptools import setup, find_packages

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name="my_package",
    version="0.1",
    include_package_data=True,
    python_requires='>=3.8',
    packages=find_packages(),
    setup_requires=['setuptools-git-versioning'],
    install_requires=requirements,
    author="Abdellah HALLOU",
    author_email="abdeallahhallou33@gmail.com",
    description="A short description of your package",
    long_description=open('README.md').read(),
    long_description_content_type="text/markdown",
    classifiers=[
        "Programming Language :: Python :: 3.8",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    version_config={
       "dirty_template": "{tag}",
    }
)

Enter fullscreen mode Exit fullscreen mode

4. Creating `requirements.txt`

In our requirements.txt file, we include the necessary dependencies for building and distributing our package:


<span>setuptools</span><span>==</span><span>69.2</span><span>.</span><span>0</span>
<span>wheel</span>
<span>twine</span>
<span>setuptools</span><span>==</span><span>69.2</span><span>.</span><span>0</span>
<span>wheel</span>
<span>twine</span>
setuptools==69.2.0
wheel
twine

Enter fullscreen mode Exit fullscreen mode

5. Building and Installing Your Package

Install the requirements. To keep things simple, we will use the Python virtual environment.


python <span>-m</span> venv <span>env source env</span>/bin/activate <span># for linux and mac</span>
./env/Scripts/activate <span># for windows</span>
pip <span>install</span> <span>-r</span> requirements.txt
python <span>-m</span> venv <span>env source env</span>/bin/activate <span># for linux and mac</span>
./env/Scripts/activate <span># for windows</span>
pip <span>install</span> <span>-r</span> requirements.txt
python -m venv env source env/bin/activate # for linux and mac
./env/Scripts/activate # for windows
pip install -r requirements.txt

Enter fullscreen mode Exit fullscreen mode

To build our package:


python setup.py sdist bdist_wheel
python setup.py sdist bdist_wheel
python setup.py sdist bdist_wheel

Enter fullscreen mode Exit fullscreen mode

To install our package locally for testing:


pip <span>install</span> <span>-e</span> <span>.</span>
pip <span>install</span> <span>-e</span> <span>.</span>
pip install -e .

Enter fullscreen mode Exit fullscreen mode

You can commit your work and ignore the folders using .gitignore file:

https://github.com/github/gitignore/blob/main/Python.gitignore

6. Publishing the Package on GitHub with a tag

To publish the package, first, create a build.pipeline.yml file at the root of the project my-package/ and commit it. The deployment will be done with twine, the library that we installed before:


<span>name</span><span>:</span> <span>Python package</span>
<span>on</span><span>:</span>
  <span>push</span><span>:</span>
    <span>tags</span><span>:</span>
      <span>-</span> <span>'</span><span>*'</span>
<span>jobs</span><span>:</span>
  <span>build</span><span>:</span>
    <span>runs-on</span><span>:</span> <span>ubuntu-latest</span>
    <span>strategy</span><span>:</span>
      <span>matrix</span><span>:</span>
        <span>python-version</span><span>:</span> <span>[</span><span>"</span><span>3.7"</span><span>,</span> <span>"</span><span>3.8"</span><span>,</span> <span>"</span><span>3.9"</span><span>,</span> <span>"</span><span>3.10"</span><span>,</span> <span>"</span><span>3.11"</span><span>]</span>
    <span>steps</span><span>:</span>
      <span>-</span> <span>uses</span><span>:</span> <span>actions/checkout@v4</span>
      <span>-</span> <span>name</span><span>:</span> <span>Set up Python </span><span>3.8</span>
        <span>uses</span><span>:</span> <span>actions/setup-python@v5</span>
        <span>with</span><span>:</span>
          <span>python-version</span><span>:</span> <span>3.8</span>
      <span>-</span> <span>name</span><span>:</span> <span>Install dependencies</span>
        <span>run</span><span>:</span> <span>|</span>
          <span>python -m pip install --upgrade pip</span>
          <span>if [ -f requirements.txt ]; then pip install -r requirements.txt; fi</span>
      <span>-</span> <span>name</span><span>:</span> <span>Build package</span>
        <span>run</span><span>:</span> <span>|</span>
          <span>python setup.py sdist bdist_wheel</span>
      <span>-</span> <span>name</span><span>:</span> <span>Publish package</span>
        <span>env</span><span>:</span>
          <span>GITHUB_TOKEN</span><span>:</span> <span>${{ secrets.GITHUB_TOKEN }}</span>
        <span>run</span><span>:</span> <span>|</span>
          <span>pip install twine</span>
          <span>python -m twine upload --repository-url https://api.github.com/orgs/${{ github.repository_owner }}/packages/pypi/upload dist/*</span>
<span>name</span><span>:</span> <span>Python package</span>

<span>on</span><span>:</span>
  <span>push</span><span>:</span>
    <span>tags</span><span>:</span>
      <span>-</span> <span>'</span><span>*'</span>

<span>jobs</span><span>:</span>
  <span>build</span><span>:</span>

    <span>runs-on</span><span>:</span> <span>ubuntu-latest</span>
    <span>strategy</span><span>:</span>
      <span>matrix</span><span>:</span>
        <span>python-version</span><span>:</span> <span>[</span><span>"</span><span>3.7"</span><span>,</span> <span>"</span><span>3.8"</span><span>,</span> <span>"</span><span>3.9"</span><span>,</span> <span>"</span><span>3.10"</span><span>,</span> <span>"</span><span>3.11"</span><span>]</span>
    <span>steps</span><span>:</span>
      <span>-</span> <span>uses</span><span>:</span> <span>actions/checkout@v4</span>
      <span>-</span> <span>name</span><span>:</span> <span>Set up Python </span><span>3.8</span>
        <span>uses</span><span>:</span> <span>actions/setup-python@v5</span>
        <span>with</span><span>:</span>
          <span>python-version</span><span>:</span> <span>3.8</span>
      <span>-</span> <span>name</span><span>:</span> <span>Install dependencies</span>
        <span>run</span><span>:</span> <span>|</span>
          <span>python -m pip install --upgrade pip</span>
          <span>if [ -f requirements.txt ]; then pip install -r requirements.txt; fi</span>
      <span>-</span> <span>name</span><span>:</span> <span>Build package</span>
        <span>run</span><span>:</span> <span>|</span>
          <span>python setup.py sdist bdist_wheel</span>
      <span>-</span> <span>name</span><span>:</span> <span>Publish package</span>
        <span>env</span><span>:</span>
          <span>GITHUB_TOKEN</span><span>:</span> <span>${{ secrets.GITHUB_TOKEN }}</span>
        <span>run</span><span>:</span> <span>|</span>
          <span>pip install twine</span>
          <span>python -m twine upload --repository-url https://api.github.com/orgs/${{ github.repository_owner }}/packages/pypi/upload dist/*</span>
name: Python package

on:
  push:
    tags:
      - '*'

jobs:
  build:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python 3.8
        uses: actions/setup-python@v5
        with:
          python-version: 3.8
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
      - name: Build package
        run: |
          python setup.py sdist bdist_wheel
      - name: Publish package
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip install twine
          python -m twine upload --repository-url https://api.github.com/orgs/${{ github.repository_owner }}/packages/pypi/upload dist/*

Enter fullscreen mode Exit fullscreen mode

If you need to include non-Python files with your module installation, you can use a MANIFEST.in file. This file specifies which additional files should be included in your package distribution.


<span># To include all files and subdirectories within the my_package/directory_name directory </span><span>recursive</span><span>-</span><span>include</span> <span>my_package</span><span>/</span><span>directory_name</span> <span>*</span>
<span># To include a specific file in your package. </span><span>include</span>  <span>my_package</span><span>/</span><span>file_name</span>
<span># To include all files and subdirectories within the my_package/directory_name directory </span><span>recursive</span><span>-</span><span>include</span> <span>my_package</span><span>/</span><span>directory_name</span> <span>*</span>
<span># To include a specific file in your package. </span><span>include</span>  <span>my_package</span><span>/</span><span>file_name</span>
# To include all files and subdirectories within the my_package/directory_name directory recursive-include my_package/directory_name *
# To include a specific file in your package. include  my_package/file_name

Enter fullscreen mode Exit fullscreen mode

Then upload the package:


git tag 0.0.1 <span>-m</span> <span>"New Release"</span>
git push origin 0.0.1
git tag 0.0.1 <span>-m</span> <span>"New Release"</span>
git push origin 0.0.1
git tag 0.0.1 -m "New Release"
git push origin 0.0.1

Enter fullscreen mode Exit fullscreen mode

7. Install the Package

Create an access token:

Go to Settings > Developer Settings > Personal access tokens (classic) > Generate new token.
Ensure you check the write:packages scope to grant the necessary permissions.

Once you have your token, keep it secure, as you’ll need it to install your package.

On your machine, you can install your private package using the following template:


<span># template</span>
pip <span>install </span>git+https://<span>{{</span> your access token <span>}}</span>@github.com/<span>{{</span> username <span>}}</span>/<span>{{</span> repository name<span>}}</span>.git@<span>{{</span> tag/version <span>}}</span><span>#egg={{ package name }}</span>
<span># example</span>
pip <span>install </span>git+https://<span>{{</span> your access token <span>}}</span>@github.com/ABDELLAH-Hallou/my-package-repo.git@0.0.1#egg<span>=</span>my_package
<span># template</span>
pip <span>install </span>git+https://<span>{{</span> your access token <span>}}</span>@github.com/<span>{{</span> username <span>}}</span>/<span>{{</span> repository name<span>}}</span>.git@<span>{{</span> tag/version <span>}}</span><span>#egg={{ package name }}</span>
<span># example</span>
pip <span>install </span>git+https://<span>{{</span> your access token <span>}}</span>@github.com/ABDELLAH-Hallou/my-package-repo.git@0.0.1#egg<span>=</span>my_package
# template
pip install git+https://{{ your access token }}@github.com/{{ username }}/{{ repository name}}.git@{{ tag/version }}#egg={{ package name }}
# example
pip install git+https://{{ your access token }}@github.com/ABDELLAH-Hallou/my-package-repo.git@0.0.1#egg=my_package

Enter fullscreen mode Exit fullscreen mode

Conclusion

Well done, you know now how to create and deploy your own private packages with Python on GitHub.

Github repository link : https://github.com/ABDELLAH-Hallou/Private-Python-Package-Deployment

原文链接：Create and Release a Private Python Package on GitHub

展开阅读全文

文章版权声明 1、本网站名称：拾光赋
2、本站永久网址：https://www.blogs.ink
3、本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ：805375623进行删除处理。
4、本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5、本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6、本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。

THE END