Pushing Python Packages to Artifact Registry Using Cloud Build

Google Artifact Registry is a powerful solution for managing and hosting Python package artifacts in a private, secure, and scalable way. This guide provides a step-by-step walkthrough to push Python package .whl files to the Artifact Registry using Google Cloud Build and a secret (creds) from Google Secret Manager for authentication.

Prerequisites

Artifact Registry Setup:

Create a Python repository in your Artifact Registry:


 gcloud artifacts repositories create python-packages <span>\</span>
   <span>--repository-format</span><span>=</span>python <span>\</span>
   <span>--location</span><span>=</span>us-central1 <span>\</span>
   <span>--description</span><span>=</span><span>"Python packages repository"</span>
 gcloud artifacts repositories create python-packages <span>\</span>
   <span>--repository-format</span><span>=</span>python <span>\</span>
   <span>--location</span><span>=</span>us-central1 <span>\</span>
   <span>--description</span><span>=</span><span>"Python packages repository"</span>
 gcloud artifacts repositories create python-packages \
   --repository-format=python \
   --location=us-central1 \
   --description="Python packages repository"

Secret Setup:

Store your key as a secret in Google Secret Manager:


 gcloud secrets create creds <span>--data-file</span><span>=</span>path/to/key.json
 gcloud secrets create creds <span>--data-file</span><span>=</span>path/to/key.json
 gcloud secrets create creds --data-file=path/to/key.json

Grant Cloud Build access to the secret:(Optional, can also be done using IAM)


 gcloud secrets add-iam-policy-binding creds <span>\</span>
   <span>--member</span><span>=</span><span>"serviceAccount:</span><span>$(</span>gcloud projects describe <span>$PROJECT_ID</span> <span>--format</span><span>=</span><span>'value(projectNumber)'</span><span>)</span><span>@cloudbuild.gserviceaccount.com"</span> <span>\</span>
   <span>--role</span><span>=</span><span>"roles/secretmanager.secretAccessor"</span>
 gcloud secrets add-iam-policy-binding creds <span>\</span>
   <span>--member</span><span>=</span><span>"serviceAccount:</span><span>$(</span>gcloud projects describe <span>$PROJECT_ID</span> <span>--format</span><span>=</span><span>'value(projectNumber)'</span><span>)</span><span>@cloudbuild.gserviceaccount.com"</span> <span>\</span>
   <span>--role</span><span>=</span><span>"roles/secretmanager.secretAccessor"</span>
 gcloud secrets add-iam-policy-binding creds \
   --member="serviceAccount:$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')@cloudbuild.gserviceaccount.com" \
   --role="roles/secretmanager.secretAccessor"

Cloud Build Permissions: Ensure your Cloud Build service account has the necessary permissions to access the Artifact Registry and Secret Manager.

Cloud Build YAML Configuration

Here’s the full working cloudbuild.yaml file:


<span>options</span><span>:</span>
  <span>machineType</span><span>:</span> <span>E2_HIGHCPU_8</span>
  <span>substitutionOption</span><span>:</span> <span>ALLOW_LOOSE</span>
  <span>logging</span><span>:</span> <span>CLOUD_LOGGING_ONLY</span>
<span>steps</span><span>:</span>
  <span># Step 1: Access the secret `creds` and save it as `key.json`</span>
  <span>-</span> <span>name</span><span>:</span> <span>'</span><span>gcr.io/google.com/cloudsdktool/cloud-sdk'</span>
    <span>entrypoint</span><span>:</span> <span>bash</span>
    <span>args</span><span>:</span>
      <span>-</span> <span>'</span><span>-c'</span>
      <span>-</span> <span>|</span>
        <span>gcloud secrets versions access latest --secret=creds > /workspace/key.json</span>
  <span># Step 2: Configure `.pypirc` with the Artifact Registry credentials</span>
  <span>-</span> <span>name</span><span>:</span> <span>'</span><span>python'</span>
    <span>entrypoint</span><span>:</span> <span>bash</span>
    <span>args</span><span>:</span>
      <span>-</span> <span>'</span><span>-c'</span>
      <span>-</span> <span>|</span>
        <span>cat > ~/.pypirc << EOL</span>
        <span>[distutils]</span>
        <span>index-servers = tower-common-repo</span>
        <span>[tower-common-repo]</span>
        <span>repository: https://us-central1-python.pkg.dev/$PROJECT_ID/python-packages/</span>
        <span>username: _json_key_base64</span>
        <span>password: $(base64 -w0 /workspace/key.json)</span>
        <span>EOL</span>
        <span># Step 3: Build and upload the Python package</span>
        <span>pip install twine build && \</span>
        <span>python -m build && \</span>
        <span>twine upload --repository tower-common-repo dist/* --verbose</span>
<span>options</span><span>:</span>
  <span>machineType</span><span>:</span> <span>E2_HIGHCPU_8</span>
  <span>substitutionOption</span><span>:</span> <span>ALLOW_LOOSE</span>
  <span>logging</span><span>:</span> <span>CLOUD_LOGGING_ONLY</span>

<span>steps</span><span>:</span>
  <span># Step 1: Access the secret `creds` and save it as `key.json`</span>
  <span>-</span> <span>name</span><span>:</span> <span>'</span><span>gcr.io/google.com/cloudsdktool/cloud-sdk'</span>
    <span>entrypoint</span><span>:</span> <span>bash</span>
    <span>args</span><span>:</span>
      <span>-</span> <span>'</span><span>-c'</span>
      <span>-</span> <span>|</span>
        <span>gcloud secrets versions access latest --secret=creds > /workspace/key.json</span>

  <span># Step 2: Configure `.pypirc` with the Artifact Registry credentials</span>
  <span>-</span> <span>name</span><span>:</span> <span>'</span><span>python'</span>
    <span>entrypoint</span><span>:</span> <span>bash</span>
    <span>args</span><span>:</span>
      <span>-</span> <span>'</span><span>-c'</span>
      <span>-</span> <span>|</span>
        <span>cat > ~/.pypirc << EOL</span>
        <span>[distutils]</span>
        <span>index-servers = tower-common-repo</span>

        <span>[tower-common-repo]</span>
        <span>repository: https://us-central1-python.pkg.dev/$PROJECT_ID/python-packages/</span>
        <span>username: _json_key_base64</span>
        <span>password: $(base64 -w0 /workspace/key.json)</span>
        <span>EOL</span>

        <span># Step 3: Build and upload the Python package</span>
        <span>pip install twine build && \</span>
        <span>python -m build && \</span>
        <span>twine upload --repository tower-common-repo dist/* --verbose</span>
options:
  machineType: E2_HIGHCPU_8
  substitutionOption: ALLOW_LOOSE
  logging: CLOUD_LOGGING_ONLY

steps:
  # Step 1: Access the secret `creds` and save it as `key.json`
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: bash
    args:
      - '-c'
      - |
        gcloud secrets versions access latest --secret=creds > /workspace/key.json

  # Step 2: Configure `.pypirc` with the Artifact Registry credentials
  - name: 'python'
    entrypoint: bash
    args:
      - '-c'
      - |
        cat > ~/.pypirc << EOL
        [distutils]
        index-servers = tower-common-repo

        [tower-common-repo]
        repository: https://us-central1-python.pkg.dev/$PROJECT_ID/python-packages/
        username: _json_key_base64
        password: $(base64 -w0 /workspace/key.json)
        EOL

        # Step 3: Build and upload the Python package
        pip install twine build && \
        python -m build && \
        twine upload --repository tower-common-repo dist/* --verbose

Enter fullscreen mode Exit fullscreen mode

Step-by-Step Explanation

Define Build Options:
- Set the machine type, substitution behavior, and logging options.
- These configurations ensure efficient builds and manageable logs.
Retrieve key.json Secret:
- Use gcloud secrets versions access to fetch the key.json file securely from Secret Manager.
- Save the file to a known location (/workspace/key.json).
Configure .pypirc:
- Generate a .pypirc file dynamically. This file is required for twine to authenticate with the Artifact Registry.
- The password is base64-encoded content of key.json.
Build and Push Package:
- Install necessary tools (twine, build).
- Build the Python package (python -m build).
- Use twine upload to push the .whl file to the Artifact Registry.

Triggering the Build

Save the cloudbuild.yaml file and trigger the build or can connect to github repository:


gcloud builds submit <span>--config</span><span>=</span>cloudbuild.yaml <span>.</span>
gcloud builds submit <span>--config</span><span>=</span>cloudbuild.yaml <span>.</span>
gcloud builds submit --config=cloudbuild.yaml .

Enter fullscreen mode Exit fullscreen mode

Key Points

Secure Secrets Management: The secret (key.json) is accessed securely using Google Secret Manager.
Dynamic Configuration: .pypirc is generated during the build, ensuring no sensitive data is stored in the repository.
Automated Upload: The process automates package building and pushing, reducing manual intervention.

Validation

After the build completes:

Verify the uploaded package in the Artifact Registry:


   gcloud artifacts packages list <span>--repository</span><span>=</span>python-packages <span>--location</span><span>=</span>us-central1
   gcloud artifacts packages list <span>--repository</span><span>=</span>python-packages <span>--location</span><span>=</span>us-central1
   gcloud artifacts packages list --repository=python-packages --location=us-central1

Enter fullscreen mode Exit fullscreen mode

Check for errors or warnings in the build logs.

原文链接：Pushing Python Packages to Artifact Registry Using Cloud Build

展开阅读全文

文章版权声明 1、本网站名称：拾光赋
2、本站永久网址：https://www.blogs.ink
3、本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ：805375623进行删除处理。
4、本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5、本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6、本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。

THE END