Deploying Python projects to air-gapped systems

Motivation

A lot of times in the real world we have to deploy Python projects behind a company firewall, restricted systems or even an air-gapped system. This is most certainly a big hassle for most people.
I’ve found a pretty good setup which allows us to develop using modern tooling like poetry, but also deploy to any system without the need to pull any packages from the internet.

Project setup

The project is setup as follows

airgap/
├── airgap/
│   ├── __init__.py
│   └── main.py
├── poetry.lock
├── pyproject.toml
├── README.md
airgap/
├── airgap/
│   ├── __init__.py
│   └── main.py
├── poetry.lock
├── pyproject.toml
├── README.md
airgap/
├── airgap/
│   ├── __init__.py
│   └── main.py
├── poetry.lock
├── pyproject.toml
├── README.md

Enter fullscreen mode Exit fullscreen mode

and we’re pulling some dependencies into our project (I’ve used arrow which is an awesome date and time manipulation library):

<span>[tool.poetry]</span>
<span>name</span> <span>=</span> <span>"airgap"</span>
<span>version</span> <span>=</span> <span>"0.1.0"</span>
<span>description</span> <span>=</span> <span>"A project running on an air-gapped system."</span>
<span>authors</span> <span>=</span> <span>["..."]</span>
<span>readme</span> <span>=</span> <span>"README.md"</span>
<span>[tool.poetry.dependencies]</span>
<span>python</span> <span>=</span> <span>"^3.10"</span>
<span>arrow</span> <span>=</span> <span>"^1.3.0"</span>
<span>[build-system]</span>
<span>requires</span> <span>=</span> <span>["poetry-core"]</span>
<span>build-backend</span> <span>=</span> <span>"poetry.core.masonry.api"</span>
<span>[tool.poetry]</span>
<span>name</span> <span>=</span> <span>"airgap"</span>
<span>version</span> <span>=</span> <span>"0.1.0"</span>
<span>description</span> <span>=</span> <span>"A project running on an air-gapped system."</span>
<span>authors</span> <span>=</span> <span>["..."]</span>
<span>readme</span> <span>=</span> <span>"README.md"</span>

<span>[tool.poetry.dependencies]</span>
<span>python</span> <span>=</span> <span>"^3.10"</span>
<span>arrow</span> <span>=</span> <span>"^1.3.0"</span>


<span>[build-system]</span>
<span>requires</span> <span>=</span> <span>["poetry-core"]</span>
<span>build-backend</span> <span>=</span> <span>"poetry.core.masonry.api"</span>
[tool.poetry]
name = "airgap"
version = "0.1.0"
description = "A project running on an air-gapped system."
authors = ["..."]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
arrow = "^1.3.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Enter fullscreen mode Exit fullscreen mode

We can have as many dependencies as we want, but for demonstration purposes I’ll keep it short.

Preparation before distribution

We’ll need to generate a requirements.txt file for our next steps. Let’s use poetry to do that:

poetry <span>export</span> <span>-f</span> requirements.txt <span>-o</span> requirements.txt
poetry <span>export</span> <span>-f</span> requirements.txt <span>-o</span> requirements.txt
poetry export -f requirements.txt -o requirements.txt

Enter fullscreen mode Exit fullscreen mode

Command breakdown:

• export the command used by poetry to convert the pyproject.toml file
• -f: the format to use, only contstraints.txt and requirements.txt supported
• -o: the name of the output file

After running you’ll get a requirements.txt similar to this one (except the hashes which I’ve omitted for brevity):

arrow==1.3.0 ; python_version >= "3.10" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.10" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.10" and python_version < "4.0"
types-python-dateutil==2.8.19.14 ; python_version >= "3.10" and python_version < "4.0"
arrow==1.3.0 ; python_version >= "3.10" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.10" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.10" and python_version < "4.0"
types-python-dateutil==2.8.19.14 ; python_version >= "3.10" and python_version < "4.0"
arrow==1.3.0 ; python_version >= "3.10" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.10" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.10" and python_version < "4.0"
types-python-dateutil==2.8.19.14 ; python_version >= "3.10" and python_version < "4.0"

Enter fullscreen mode Exit fullscreen mode

Now comes the funky part. We’ll create wheels from these dependencies and store them locally. We’ll use pip to get the wheels:

pip wheel <span>--no-deps</span> <span>--wheel-dir</span> ./wheels <span>-r</span> requirements.txt
pip wheel <span>--no-deps</span> <span>--wheel-dir</span> ./wheels <span>-r</span> requirements.txt
pip wheel --no-deps --wheel-dir ./wheels -r requirements.txt

Enter fullscreen mode Exit fullscreen mode

Command breakdown:

• wheel: the pip command used to generate wheels docs
• --no-deps: do not install dependencies (we’ve already got them all in the requirements.txt)
• --wheel-dir: the directory to store the wheels (doesn’t have to exist, it will be created for you)
• -r: which requirements file to use

Our requirements.txt includes hashes, which means that pip wheel will imply the --require-hashes option. In turn this will verify the packages when installing.

Now our directory wheels will hold four packages (the ones from the requirements.txt). The final part is our own package source. We’ll use poetry to build the wheel and place it in the wheels directory.

poetry build <span>&&</span> <span>mv </span>dist/<span>*</span>.whl ./wheels
poetry build <span>&&</span> <span>mv </span>dist/<span>*</span>.whl ./wheels
poetry build && mv dist/*.whl ./wheels

Enter fullscreen mode Exit fullscreen mode

Command breakdown:

• build: the poetry command which builds our project
• mv: move files
• dist/*.whl: source all files ending in .whl in the dist/ directory
• ./wheels: the destination for the sourced files

At this point we’re done with the preparation. Just zip the ./wheels directory and distribute it to your air-gapped server. We’ll just create a zip of the files:

zip <span>-r</span> airgap-dist-0.1.0.zip ./wheels/<span>*</span>
zip <span>-r</span> airgap-dist-0.1.0.zip ./wheels/<span>*</span>
zip -r airgap-dist-0.1.0.zip ./wheels/*

Enter fullscreen mode Exit fullscreen mode

Command breakdown:

• -r: compress the archive
• iargap-dist-0.1.0.zip: name of the compressed archive
• ./wheels/*: the sources to put in the archive

Deployment

Get your zip to your air-gapped system, unpack it and install. The best-practice I stick to is to use the /opt directory to deploy custom software. Let’s put it there:

<span>mkdir</span> /opt/airgap
unzip airgap-dist-0.1.0.zip <span>-d</span> /opt/airgap
<span>mkdir</span> /opt/airgap
unzip airgap-dist-0.1.0.zip <span>-d</span> /opt/airgap
mkdir /opt/airgap
unzip airgap-dist-0.1.0.zip -d /opt/airgap

Enter fullscreen mode Exit fullscreen mode

Note: you might have to run the mkdir command using sudo. It’s very dependant on your setup, user, OS version etc.

Just as an example you might need to do this:

<span>sudo mkdir</span> /opt/airgap
<span>sudo chown </span>youruser:yourgroup /opt/airgap
<span>sudo chmod </span>755 /opt/airgap
<span>sudo mkdir</span> /opt/airgap
<span>sudo chown </span>youruser:yourgroup /opt/airgap
<span>sudo chmod </span>755 /opt/airgap
sudo mkdir /opt/airgap
sudo chown youruser:yourgroup /opt/airgap
sudo chmod 755 /opt/airgap

Enter fullscreen mode Exit fullscreen mode

Now in general I wouldn’t advocate to install any project in the global Python site, but since we’re striving for simplicity we’ll do that. The next step is very easy, install our wheels:

pip <span>install</span> <span>--no-cache</span> /opt/airgap/wheels/<span>*</span>
pip <span>install</span> <span>--no-cache</span> /opt/airgap/wheels/<span>*</span>
pip install --no-cache /opt/airgap/wheels/*

Enter fullscreen mode Exit fullscreen mode

Bonus

Just in case you want to use venv you can do the following:

python <span>-m</span> venv /opt/airgap/venv
python <span>-m</span> venv /opt/airgap/venv
python -m venv /opt/airgap/venv

Enter fullscreen mode Exit fullscreen mode

Command breakdown:

• -m venv: tell Python to use the module venv (should be installed by default)
• /opt/airgap/venv: the location where to create the virtual environment, note that the directory name venv is just a convention, but you can use any name you like.

Activate the newly created virtual environment:

<span>source</span> /opt/airgap/venv/bin/activate
<span>source</span> /opt/airgap/venv/bin/activate
source /opt/airgap/venv/bin/activate

Enter fullscreen mode Exit fullscreen mode

And then do the install:

<span># verify you're using the venv pip</span>
which pip
<span>$ </span>/opt/airgap/venv/bin/pip
<span># install your packages</span>
pip <span>install</span> <span>--no-cache</span> /opt/airgap/wheels/<span>*</span>
<span># verify you're using the venv pip</span>
which pip
<span>$ </span>/opt/airgap/venv/bin/pip

<span># install your packages</span>
pip <span>install</span> <span>--no-cache</span> /opt/airgap/wheels/<span>*</span>
# verify you're using the venv pip
which pip
$ /opt/airgap/venv/bin/pip

# install your packages
pip install --no-cache /opt/airgap/wheels/*

Enter fullscreen mode Exit fullscreen mode

Note: to exit the virtual environment you can run deactivate in your terminal. It’s a command automatically available as soon as you activate a virtual environment

Conclusion

We’ve seen how to extract the dependencies from our new pyproject.toml file, build wheels and zip them up for distribution. Consequently how we can deploy our project to our air-gapped system. Happy coding!

原文链接：Deploying Python projects to air-gapped systems