Originally published on my site at writingco.de. This is a cross-post from my content blog. I will be publishing new content every week or so, and you can sign up to my newsletter if you’d like to receive my articles and other info tidbits directly to your inbox!
There is a history of confusion around packaging and managing a python virtual environment. Do we use pyvenv? venv? virtualenvwrapper? Lately, other options have come out of the woodwork. The use of pyenv gives us the ability to manage python versions much like nvm and rvm, while Pipenv is the successor to pip itself and is slated to be merged in eventually.
Installing Pyenv
Pyenv allows you to easily install different versions of python along side each other on your system without conflicts. It is not supported on windows but python can easily be installed from their website. Just make sure to add it to your windows PATH.
Installation is pretty straight forward, simply run these commands to get it set up.
$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv $ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile $ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv $ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile $ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
Enter fullscreen mode Exit fullscreen mode
There are some more installation details on their repo page if you have any trouble. They also have instructions for users of zsh. You will likely need to reload your terminal after installation to have the pyenv command available.
pyenvpyenv 1.2.12-2-geb68ec94Usage: pyenv <command> [<args>]Some useful pyenv commands are:commands List all available pyenv commandslocal Set or show the local application-specific Python versionglobal Set or show the global Python versionshell Set or show the shell-specific Python versioninstall Install a Python version using python-builduninstall Uninstall a specific Python versionrehash Rehash pyenv shims (run this after installing executables)version Show the current Python version and its originversions List all Python versions available to pyenvwhich Display the full path to an executablewhence List all Python versions that contain the given executableSee `pyenv help <command>' for information on a specific command.For full documentation, see: https://github.com/pyenv/pyenv#readmepyenv pyenv 1.2.12-2-geb68ec94 Usage: pyenv <command> [<args>] Some useful pyenv commands are: commands List all available pyenv commands local Set or show the local application-specific Python version global Set or show the global Python version shell Set or show the shell-specific Python version install Install a Python version using python-build uninstall Uninstall a specific Python version rehash Rehash pyenv shims (run this after installing executables) version Show the current Python version and its origin versions List all Python versions available to pyenv which Display the full path to an executable whence List all Python versions that contain the given executable See `pyenv help <command>' for information on a specific command. For full documentation, see: https://github.com/pyenv/pyenv#readmepyenv pyenv 1.2.12-2-geb68ec94 Usage: pyenv <command> [<args>] Some useful pyenv commands are: commands List all available pyenv commands local Set or show the local application-specific Python version global Set or show the global Python version shell Set or show the shell-specific Python version install Install a Python version using python-build uninstall Uninstall a specific Python version rehash Rehash pyenv shims (run this after installing executables) version Show the current Python version and its origin versions List all Python versions available to pyenv which Display the full path to an executable whence List all Python versions that contain the given executable See `pyenv help <command>' for information on a specific command. For full documentation, see: https://github.com/pyenv/pyenv#readme
Enter fullscreen mode Exit fullscreen mode
The primary sub-commands we will be dealing with today are local, global, and install. First, let’s look and see what versions we have on our system.
pyenv versions* systempyenv versions * systempyenv versions * system
Enter fullscreen mode Exit fullscreen mode
Installing New Versions
For most users, you will only have the system interpreter. Let’s install 3.7 as another global version of python.
pyenv install 3.7python-build: definition not found: 3.7The following versions contain `3.7' in the name:2.3.73.3.73.7.03.7-dev3.7.13.7.23.7.3miniconda-3.7.0miniconda3-3.7.0stackless-3.3.7See all available versions with `pyenv install --list'....pyenv install 3.7 python-build: definition not found: 3.7 The following versions contain `3.7' in the name: 2.3.7 3.3.7 3.7.0 3.7-dev 3.7.1 3.7.2 3.7.3 miniconda-3.7.0 miniconda3-3.7.0 stackless-3.3.7 See all available versions with `pyenv install --list'. ...pyenv install 3.7 python-build: definition not found: 3.7 The following versions contain `3.7' in the name: 2.3.7 3.3.7 3.7.0 3.7-dev 3.7.1 3.7.2 3.7.3 miniconda-3.7.0 miniconda3-3.7.0 stackless-3.3.7 See all available versions with `pyenv install --list'. ...
Enter fullscreen mode Exit fullscreen mode
If the version we ask for is only a partial match, it will show us which versions we can install. Let’s install the latest, which will take a few minutes.
pyenv install 3.7.3Downloading Python-3.7.3.tar.xz...-> https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xzInstalling Python-3.7.3...Installed Python-3.7.3 to /home/shawn/.pyenv/versions/3.7.3pyenv install 3.7.3 Downloading Python-3.7.3.tar.xz... -> https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz Installing Python-3.7.3... Installed Python-3.7.3 to /home/shawn/.pyenv/versions/3.7.3pyenv install 3.7.3 Downloading Python-3.7.3.tar.xz... -> https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz Installing Python-3.7.3... Installed Python-3.7.3 to /home/shawn/.pyenv/versions/3.7.3
Enter fullscreen mode Exit fullscreen mode
Now let’s make 3.7.3 globally available so there is no messing with our system python:
pyenv globalsystempyenv global 3.7.3pyenv global3.7.3pyenv versionssystem* 3.7.3 (set by /home/shawn/.pyenv/version)pyenv global system pyenv global 3.7.3 pyenv global 3.7.3 pyenv versions system * 3.7.3 (set by /home/shawn/.pyenv/version)pyenv global system pyenv global 3.7.3 pyenv global 3.7.3 pyenv versions system * 3.7.3 (set by /home/shawn/.pyenv/version)
Enter fullscreen mode Exit fullscreen mode
We are now defaulting to 3.7.3. For now, we are done with pyenv. It is a very simple tool to use.
Pipenv: the Python Virtual Environment
To install pipenv, you can install it through your system’s package manager as described here on their repo page.
The setup
To get started we need to create a simple project. Let’s clone a repo I have that has a few basic files in it for tracking projects in git.
git clone git@gitlab.com:autoferrit/template.git demoCloning into 'demo'...remote: Enumerating objects: 6, done.remote: Counting objects: 100% (6/6), done.remote: Compressing objects: 100% (6/6), done.remote: Total 6 (delta 0), reused 0 (delta 0)Receiving objects: 100% (6/6), done.cd demols -laPermissions Size User Date Modified Git Namedrwxr-xr-x - shawn 27 Jun 12:10 -- .git.rwxr-xr-x 474 shawn 27 Jun 12:09 -- .editorconfig.rwxr-xr-x 2.2k shawn 27 Jun 12:09 -- .gitattributes.rw-r--r-- 138 shawn 27 Jun 12:09 -- Pipfile.rw-r--r-- 96 shawn 27 Jun 12:09 -- Readme.mdgit clone git@gitlab.com:autoferrit/template.git demo Cloning into 'demo'... remote: Enumerating objects: 6, done. remote: Counting objects: 100% (6/6), done. remote: Compressing objects: 100% (6/6), done. remote: Total 6 (delta 0), reused 0 (delta 0) Receiving objects: 100% (6/6), done. cd demo ls -la Permissions Size User Date Modified Git Name drwxr-xr-x - shawn 27 Jun 12:10 -- .git .rwxr-xr-x 474 shawn 27 Jun 12:09 -- .editorconfig .rwxr-xr-x 2.2k shawn 27 Jun 12:09 -- .gitattributes .rw-r--r-- 138 shawn 27 Jun 12:09 -- Pipfile .rw-r--r-- 96 shawn 27 Jun 12:09 -- Readme.mdgit clone git@gitlab.com:autoferrit/template.git demo Cloning into 'demo'... remote: Enumerating objects: 6, done. remote: Counting objects: 100% (6/6), done. remote: Compressing objects: 100% (6/6), done. remote: Total 6 (delta 0), reused 0 (delta 0) Receiving objects: 100% (6/6), done. cd demo ls -la Permissions Size User Date Modified Git Name drwxr-xr-x - shawn 27 Jun 12:10 -- .git .rwxr-xr-x 474 shawn 27 Jun 12:09 -- .editorconfig .rwxr-xr-x 2.2k shawn 27 Jun 12:09 -- .gitattributes .rw-r--r-- 138 shawn 27 Jun 12:09 -- Pipfile .rw-r--r-- 96 shawn 27 Jun 12:09 -- Readme.md
Enter fullscreen mode Exit fullscreen mode
You can see the file contents at the GitLab project.
-
.editorconfig: helps tell the editors (with appropriate plugins) what formatting options should be used on files in this project. -
.gitattributes: This keeps line endings in check when working on various platforms (I am looking at you, windows). -
PipfileThis is where we will keep of application requirements rather than using requirements.txt
The Pipfile would also be generated from the following command.
pipenv --python 3.7.3 installpipenv --python 3.7.3 installpipenv --python 3.7.3 install
Enter fullscreen mode Exit fullscreen mode
This will create your python virtual environment, which you can then use by running pipenv shell. The best part is that Pipenv supports Pyenv and will use python versions installed using that tool making our system much cleaner.
A simple API
To show how to use it, we are going to create a very basic API using FastAPI. Install the requirements with the following command:
pipenv install fastapi uvicornpipenv install fastapi uvicornpipenv install fastapi uvicorn
Enter fullscreen mode Exit fullscreen mode
This will create a Pipfile.lock which is analogous to package-lock.json in node.js. Creating a repeatable environment when the code is run through a CI server or production is critical and this is where Pipenv shines. It makes sure we can actually deploy the same versions of packages AND their requirements. Let’s run this code.
uvicorn main:app --reloaduvicorn main:app --reloaduvicorn main:app --reload
Enter fullscreen mode Exit fullscreen mode
Now you can see your API by visiting http://127.0.0.1:8000/items/5?q=somequery. If you would like to follow along in this tutorial, you can read the fastapi tutorial in their documentation here.
The Pipfile
Let’s open up our Pipfile to see what has installed in our python virtual environment.
[[source]]name = "pypi"url = "https://pypi.org/simple"verify_ssl = true[dev-packages][packages]fastapi = "*"uvicorn = "*"[requires]python_version = "3.7"[[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] fastapi = "*" uvicorn = "*" [requires] python_version = "3.7"[[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] fastapi = "*" uvicorn = "*" [requires] python_version = "3.7"
Enter fullscreen mode Exit fullscreen mode
By default, Pipenv will always install the newest versions of packages. Which is what we should all be doing. When new versions are released and we run pipenv install it will install the newer version that was released. To update a package, we can run pipenv update fastapi or if we want to update all we can run pipenv update. If we need to pin a specific version, you can do so in the same way if you were to use standard pip and requirements.txt.
How do I install Dev Packages in my environment?
To install development packages just pass the --dev argument.
pipenv install --dev ptpythonpipenv install --dev ptpythonpipenv install --dev ptpython
Enter fullscreen mode Exit fullscreen mode
And to install development packages:
pipenv install --devpipenv install --devpipenv install --dev
Enter fullscreen mode Exit fullscreen mode
By default, pipenv install will ONLY install the base packages that should live on production. This is what we want. Passing the --dev parameter will allow us to install base AND dev packages locally or during the CI process.
The pip in the $SHELL
To activate the environment, simply run
demo on writingcode master [!?]pipenv shellLaunching subshell in virtual environment…. /home/shawn/code/sandbox/demo/.venv/bin/activatedemo on writingcode master [!?] via demopip listPackage Version---------- -------Click 7.0fastapi 0.30.0h11 0.8.1httptools 0.0.13pip 19.1.1pydantic 0.28setuptools 41.0.1starlette 0.12.0uvicorn 0.8.2uvloop 0.12.2websockets 7.0wheel 0.33.4demo on writingcode master [!?] pipenv shell Launching subshell in virtual environment… . /home/shawn/code/sandbox/demo/.venv/bin/activate demo on writingcode master [!?] via demo pip list Package Version ---------- ------- Click 7.0 fastapi 0.30.0 h11 0.8.1 httptools 0.0.13 pip 19.1.1 pydantic 0.28 setuptools 41.0.1 starlette 0.12.0 uvicorn 0.8.2 uvloop 0.12.2 websockets 7.0 wheel 0.33.4demo on writingcode master [!?] pipenv shell Launching subshell in virtual environment… . /home/shawn/code/sandbox/demo/.venv/bin/activate demo on writingcode master [!?] via demo pip list Package Version ---------- ------- Click 7.0 fastapi 0.30.0 h11 0.8.1 httptools 0.0.13 pip 19.1.1 pydantic 0.28 setuptools 41.0.1 starlette 0.12.0 uvicorn 0.8.2 uvloop 0.12.2 websockets 7.0 wheel 0.33.4
Enter fullscreen mode Exit fullscreen mode
Note: I am showing my full terminal text here to show that it is now adding via demo to the prompt for the python environment. You will likely see this represented differently. Often times it looks like this:
(demo) $ pip list...(demo) $ pip list ...(demo) $ pip list ...
Enter fullscreen mode Exit fullscreen mode
Checking for vulnerabilities in your virtual environment
Possibly one of the best features is the ability to check for vulnerabilities.
pipenv checkChecking PEP 508 requirements…Passed!Checking installed package safety…All good!pipenv check Checking PEP 508 requirements… Passed! Checking installed package safety… All good!pipenv check Checking PEP 508 requirements… Passed! Checking installed package safety… All good!
Enter fullscreen mode Exit fullscreen mode
We have no vulnerabilities in our python virtual environment. That’s a good thing. Let’s try to add one.
pipenv install 'django==2.1'pipenv checkChecking PEP 508 requirements…Passed!Checking installed package safety…36883: django <2.1.6,>=2.1.0 resolved (2.1 installed)!Django 2.1.x before 2.1.6 allows Uncontrolled Memory Consumption via a malicious attacker-supplied value to the django.utils.numberformat.format() function.36522: django <2.1.2,>=2.1 resolved (2.1 installed)!An issue was discovered in Django 2.1 before 2.1.2, in which unprivileged users can read the password hashes of arbitrary accounts. The read-only password widget used by the Django Admin to display an obfuscated password hash was bypassed if a user has only the "view" permission (new in Django 2.1), resulting in display of the entire password hash to those users. This may result in a vulnerability for sites with legacy user accounts using insecure hashes.36517: django <2.1.2,>=2.1.0 resolved (2.1 installed)!django before 2.1.2 fixes a security bug in 2.1.x.If an admin user has the change permission to the user model, only part of thepassword hash is displayed in the change form. Admin users with the view (butnot change) permission to the user model were displayed the entire hash.pipenv install 'django==2.1' pipenv check Checking PEP 508 requirements… Passed! Checking installed package safety… 36883: django <2.1.6,>=2.1.0 resolved (2.1 installed)! Django 2.1.x before 2.1.6 allows Uncontrolled Memory Consumption via a malicious attacker-supplied value to the django.utils.numberformat.format() function. 36522: django <2.1.2,>=2.1 resolved (2.1 installed)! An issue was discovered in Django 2.1 before 2.1.2, in which unprivileged users can read the password hashes of arbitrary accounts. The read-only password widget used by the Django Admin to display an obfuscated password hash was bypassed if a user has only the "view" permission (new in Django 2.1), resulting in display of the entire password hash to those users. This may result in a vulnerability for sites with legacy user accounts using insecure hashes. 36517: django <2.1.2,>=2.1.0 resolved (2.1 installed)! django before 2.1.2 fixes a security bug in 2.1.x. If an admin user has the change permission to the user model, only part of the password hash is displayed in the change form. Admin users with the view (but not change) permission to the user model were displayed the entire hash.pipenv install 'django==2.1' pipenv check Checking PEP 508 requirements… Passed! Checking installed package safety… 36883: django <2.1.6,>=2.1.0 resolved (2.1 installed)! Django 2.1.x before 2.1.6 allows Uncontrolled Memory Consumption via a malicious attacker-supplied value to the django.utils.numberformat.format() function. 36522: django <2.1.2,>=2.1 resolved (2.1 installed)! An issue was discovered in Django 2.1 before 2.1.2, in which unprivileged users can read the password hashes of arbitrary accounts. The read-only password widget used by the Django Admin to display an obfuscated password hash was bypassed if a user has only the "view" permission (new in Django 2.1), resulting in display of the entire password hash to those users. This may result in a vulnerability for sites with legacy user accounts using insecure hashes. 36517: django <2.1.2,>=2.1.0 resolved (2.1 installed)! django before 2.1.2 fixes a security bug in 2.1.x. If an admin user has the change permission to the user model, only part of the password hash is displayed in the change form. Admin users with the view (but not change) permission to the user model were displayed the entire hash.
Enter fullscreen mode Exit fullscreen mode
WHOAH! Don’t install Django 2.1. Luckily it was fixed, let’s update it.
pipenv update djangoLocking [dev-packages] dependencies…Locking [packages] dependencies…Success!Updated Pipfile.lock (9d4a23)!Installing dependencies from Pipfile.lock (9d4a23)…▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 11/11 — 00:00:01All dependencies are now up-to-date!pip listPackage Version---------- -------...Django 2.1...pipenv update django Locking [dev-packages] dependencies… Locking [packages] dependencies… Success! Updated Pipfile.lock (9d4a23)! Installing dependencies from Pipfile.lock (9d4a23)… ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 11/11 — 00:00:01 All dependencies are now up-to-date! pip list Package Version ---------- ------- ... Django 2.1 ...pipenv update django Locking [dev-packages] dependencies… Locking [packages] dependencies… Success! Updated Pipfile.lock (9d4a23)! Installing dependencies from Pipfile.lock (9d4a23)… ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 11/11 — 00:00:01 All dependencies are now up-to-date! pip list Package Version ---------- ------- ... Django 2.1 ...
Enter fullscreen mode Exit fullscreen mode
Wait why is the old version installed? This is GOOD! Why? It means that our setup is idempotent. Why? It didn’t work because we pinned version “2.1” of Django. So we should unpin that in the Pipfile by changing the version to "*". We should now be able to have the expected outcome.
pipenv installPipfile.lock (10cb8d) out of date, updating to (9d4a23)…Locking [dev-packages] dependencies…Locking [packages] dependencies…Success!Updated Pipfile.lock (10cb8d)!Installing dependencies from Pipfile.lock (10cb8d)…▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 12/12 — 00:00:04pip listPackage Version---------- -------...Django 2.2.2...pipenv install Pipfile.lock (10cb8d) out of date, updating to (9d4a23)… Locking [dev-packages] dependencies… Locking [packages] dependencies… Success! Updated Pipfile.lock (10cb8d)! Installing dependencies from Pipfile.lock (10cb8d)… ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 12/12 — 00:00:04 pip list Package Version ---------- ------- ... Django 2.2.2 ...pipenv install Pipfile.lock (10cb8d) out of date, updating to (9d4a23)… Locking [dev-packages] dependencies… Locking [packages] dependencies… Success! Updated Pipfile.lock (10cb8d)! Installing dependencies from Pipfile.lock (10cb8d)… ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 12/12 — 00:00:04 pip list Package Version ---------- ------- ... Django 2.2.2 ...
Enter fullscreen mode Exit fullscreen mode
Conclusion
Pyenv and Pipenv make it much easier to keep your python virtual environment and versions in check. You can now make your environments consistently deployable across systems. Python has a history of being a headache when it comes to consistently installing requirements across systems. Using these tools will help you focus more on your actual code.
Unfortunately, Pipenv isn’t geared towards those making libraries to be uploaded to pypi which still requires setup.py. Hopefully, in the future, they will find a way to update the packaging system with these too
And Here is the link again to subscribe to my newsletter. You can also follow me on twitter.
How do you manage your environments and dependencies in Python?
暂无评论内容