LGTM Devlog 11: Writing the Serverless Function for receiving GitHub webhooks with Pydantic validation

LGTM (40 Part Series)

1 LGTM Devlog 0: Teaching Git through playing a game
2 LGTM Devlog 1: New Logo, and jumping the gun by buying a domain
36 more parts…
3 LGTM Devlog 2: Design Ideas for the game I
4 LGTM Devlog 3: Design Ideas for the game 2
5 LGTM Devlog 4: Technical Decisions – GitHub API and Python
6 LGTM Devlog 5: Technical Decisions – Serverless Architecture
7 LGTM Devlog 6: Sprint 1 plans
8 LGTM Devlog 7: Creating a Firebase project
9 LGTM Devlog 8: Starting a git repo for an open source project from scratch
10 LGTM Devlog 9: Python Google Cloud Functions with Unit Tests and Linting
11 LGTM Devlog 10: Capturing the GitHub webhook for fork requests
12 LGTM Devlog 11: Writing the Serverless Function for receiving GitHub webhooks with Pydantic validation
13 LGTM Devlog 12: CI/CD with GitHub Actions to run Unit Tests and deploy Firebase Functions
14 LGTM Devlog 13: GitHub Branch Protection and Security
15 LGTM Devlog 14: Sprint 1 Retrospective
16 LGTM Devlog 15: Sprint 2 plans
17 LGTM Devlog 16: A serverless data base access rule conundrum
18 LGTM Devlog 17: Website and GitHub OAuth
19 LGTM Devlog 18: Python Serverless functions using GitHub API to validate users
20 LGTM Devlog 19: Game data/quest storage
21 LGTM Devlog 20: Python Abstract Base Class-based data/quest storage
22 LGTM Devlog 21: Deploying Pub/Sub-triggered Python Google Cloud Functions
23 LGTM Devlog 22: Modularization
24 LGTM Devlog 23: Sprint 2 Retrospective
25 LGTM Devlog 24: Sprint 3 plans
26 LGTM Devlog 25: Some cleanup
27 LGTM Devlog 26: Python Graphlib DAGs for Quest Stages
28 LGTM Devlog 27: Branching quests
29 LGTM Devlog 28: Game event loop using Google Cloud Scheduler and PubSub
30 LGTM Devlog 29: ORM for Firestore and __init__subclass__ dunders and metaclasses
31 LGTM Devlog 30: Sprint 3 Retrospective
32 LGTM Devlog 31: Sprint 4 Plan
33 LGTM Devlog 32: Secrets Management to avoid storing API keys in services
34 LGTM Devlog 33: Using PyGithub to post GitHub issues and comments
35 LGTM Devlog 34: Characters Posting on GitHub Issues
36 LGTM Devlog 35: Responding to the player’s answers on GitHub Issues Comments
37 LGTM Devlog 36: Character profiles! On GitHub
38 LGTM Devlog 37: Sprint 4 Retrospective
39 LGTM Devlog 38: Sprint 5 Plan
40 LGTM Devlog 39: Planning the story

Now that we know what the github forks look like, it’s time to write the function for receiving them. You can see the code matching this post at this commit: 40ac1367

Data validation with Pydantic

First, validating data and extracting it. I need to pull out four or five pieces of data from the json being sent. While this is a relatively simple case of reading the various nested keys from the json, there’s problems with that: what if some of the keys don’t exist? We need to throw an error, ideally one that explains what the problem is. Suddenly pulling out 4 values becomes having to write four if statements and for error messages.

While in this particular instance, we don’t expect the github API to be omitting values, it’s still to do things rigorously. One of the best options for this validation problem is to use a module like Pydantic. Not only does it do this validation for you, it generates meaningful errors, has static type annotations, and a whole bunch of very wholesome reasons why you’d use Pydantic over manually decoding JSONs.

In fact, it’s so useful that Python API libraries like Starlette/FastAPI built-in support for Pydantic models for defining what API endpoints require as payload, or output. Because we’re running on Google’s serverless functions, we don’t have those built-in support for Pydantic, but we can still make use of it, we just have to manually run the validation.

So first, we define the data model(s). I noticed that the hook json payload has two members: forkee and repository. They both describe a repo. And inside each repo, is an owner that contains the details of the person who owns the repo. So I decided to buidl my data models like this:

(This is the entire contents of the app/utils/models.py file)

<span>"""</span><span> Models for validation of Github hooks </span><span>"""</span>
<span>from</span> <span>pydantic</span> <span>import</span> <span>BaseModel</span><span>,</span> <span>Field</span><span>,</span> <span>Extra</span> <span># pylint: disable=no-name-in-module </span>
<span># pylint: disable=too-few-public-methods,missing-class-docstring </span><span>class</span> <span>GitHubUser</span><span>(</span><span>BaseModel</span><span>):</span>
<span>login</span><span>:</span> <span>str</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>User</span><span>'</span><span>s user login name</span><span>"</span><span>)</span>
<span>id</span><span>:</span> <span>int</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>User</span><span>'</span><span>s ID</span><span>"</span><span>)</span>
<span>class</span> <span>Config</span><span>:</span>
<span>extra</span> <span>=</span> <span>Extra</span><span>.</span><span>ignore</span>
<span>class</span> <span>GitHubRepository</span><span>(</span><span>BaseModel</span><span>):</span>
<span>id</span><span>:</span> <span>int</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>Repo</span><span>'</span><span>s ID</span><span>"</span><span>)</span>
<span>full_name</span><span>:</span> <span>str</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>Repo</span><span>'</span><span>s full name (including owner)</span><span>"</span><span>)</span>
<span>owner</span><span>:</span> <span>GitHubUser</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>Owner of repo</span><span>"</span><span>)</span>
<span>url</span><span>:</span> <span>str</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>API RUL of the repo</span><span>"</span><span>)</span>
<span>class</span> <span>Config</span><span>:</span>
<span>extra</span> <span>=</span> <span>Extra</span><span>.</span><span>ignore</span>
<span>class</span> <span>GitHubHookFork</span><span>(</span><span>BaseModel</span><span>):</span>
<span>forkee</span><span>:</span> <span>GitHubRepository</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>The fork created</span><span>"</span><span>)</span>
<span>repository</span><span>:</span> <span>GitHubRepository</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>The repository being forked</span><span>"</span><span>)</span>
<span>class</span> <span>Config</span><span>:</span>
<span>extra</span> <span>=</span> <span>Extra</span><span>.</span><span>ignore</span>
<span>"""</span><span> Models for validation of Github hooks </span><span>"""</span>
<span>from</span> <span>pydantic</span> <span>import</span> <span>BaseModel</span><span>,</span> <span>Field</span><span>,</span> <span>Extra</span>  <span># pylint: disable=no-name-in-module </span>
<span># pylint: disable=too-few-public-methods,missing-class-docstring </span><span>class</span> <span>GitHubUser</span><span>(</span><span>BaseModel</span><span>):</span>
    <span>login</span><span>:</span> <span>str</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>User</span><span>'</span><span>s user login name</span><span>"</span><span>)</span>
    <span>id</span><span>:</span> <span>int</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>User</span><span>'</span><span>s ID</span><span>"</span><span>)</span>

    <span>class</span> <span>Config</span><span>:</span>
        <span>extra</span> <span>=</span> <span>Extra</span><span>.</span><span>ignore</span>


<span>class</span> <span>GitHubRepository</span><span>(</span><span>BaseModel</span><span>):</span>
    <span>id</span><span>:</span> <span>int</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>Repo</span><span>'</span><span>s ID</span><span>"</span><span>)</span>
    <span>full_name</span><span>:</span> <span>str</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>Repo</span><span>'</span><span>s full name (including owner)</span><span>"</span><span>)</span>
    <span>owner</span><span>:</span> <span>GitHubUser</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>Owner of repo</span><span>"</span><span>)</span>
    <span>url</span><span>:</span> <span>str</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>API RUL of the repo</span><span>"</span><span>)</span>

    <span>class</span> <span>Config</span><span>:</span>
        <span>extra</span> <span>=</span> <span>Extra</span><span>.</span><span>ignore</span>


<span>class</span> <span>GitHubHookFork</span><span>(</span><span>BaseModel</span><span>):</span>
    <span>forkee</span><span>:</span> <span>GitHubRepository</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>The fork created</span><span>"</span><span>)</span>
    <span>repository</span><span>:</span> <span>GitHubRepository</span> <span>=</span> <span>Field</span><span>(...,</span> <span>title</span><span>=</span><span>"</span><span>The repository being forked</span><span>"</span><span>)</span>

    <span>class</span> <span>Config</span><span>:</span>
        <span>extra</span> <span>=</span> <span>Extra</span><span>.</span><span>ignore</span>
""" Models for validation of Github hooks """ from pydantic import BaseModel, Field, Extra # pylint: disable=no-name-in-module # pylint: disable=too-few-public-methods,missing-class-docstring class GitHubUser(BaseModel): login: str = Field(..., title="User's user login name") id: int = Field(..., title="User's ID") class Config: extra = Extra.ignore class GitHubRepository(BaseModel): id: int = Field(..., title="Repo's ID") full_name: str = Field(..., title="Repo's full name (including owner)") owner: GitHubUser = Field(..., title="Owner of repo") url: str = Field(..., title="API RUL of the repo") class Config: extra = Extra.ignore class GitHubHookFork(BaseModel): forkee: GitHubRepository = Field(..., title="The fork created") repository: GitHubRepository = Field(..., title="The repository being forked") class Config: extra = Extra.ignore

Enter fullscreen mode Exit fullscreen mode

The extra config tells Pydantic to not error out when it receives more data than was given, since I don’t want to fully define the github object, just the objects I want to access later.

Now in my main function, I can tell it to parse the raw data from the request and return this object:

<span>try</span><span>:</span>
<span>hook_fork</span> <span>=</span> <span>GitHubHookFork</span><span>.</span><span>parse_raw</span><span>(</span><span>request</span><span>.</span><span>data</span><span>)</span>
<span>except</span> <span>ValidationError</span> <span>as</span> <span>err</span><span>:</span>
<span>logger</span><span>.</span><span>err</span><span>(</span><span>"</span><span>Validation error</span><span>"</span><span>,</span> <span>err</span><span>=</span><span>err</span><span>)</span>
<span>return</span> <span>abort</span><span>(</span><span>400</span><span>,</span> <span>"</span><span>Validation error</span><span>"</span><span>)</span>
    <span>try</span><span>:</span>
        <span>hook_fork</span> <span>=</span> <span>GitHubHookFork</span><span>.</span><span>parse_raw</span><span>(</span><span>request</span><span>.</span><span>data</span><span>)</span>
    <span>except</span> <span>ValidationError</span> <span>as</span> <span>err</span><span>:</span>
        <span>logger</span><span>.</span><span>err</span><span>(</span><span>"</span><span>Validation error</span><span>"</span><span>,</span> <span>err</span><span>=</span><span>err</span><span>)</span>
        <span>return</span> <span>abort</span><span>(</span><span>400</span><span>,</span> <span>"</span><span>Validation error</span><span>"</span><span>)</span>
try: hook_fork = GitHubHookFork.parse_raw(request.data) except ValidationError as err: logger.err("Validation error", err=err) return abort(400, "Validation error")

Enter fullscreen mode Exit fullscreen mode

This hook_fork object has the properties we defined, for example, we can check the username of the person who forked our repo using hook_fork.forkee.owner.login, and! in the case validation fails, we’ll get an exception containing what part of the data didn’t match.

Signature validation

We discovered in the last post that GitHub can secure webhooks using a signature that is calculated from a pre-shared secret value that you tell it to use, and the payload. I found an example of this implementation on Google, but updated it to use the new SHA256 which GitHub wants you to use. I’m going to store the pre-shared secret in an environmental variable called SECRET (may rename later), so the code looks like this (contents of app/utils/verify.py):

<span>"""</span><span> Verify the GitHub webhook secret </span><span>"""</span>
<span>import</span> <span>os</span>
<span>import</span> <span>hmac</span>
<span>import</span> <span>hashlib</span>
<span>from</span> <span>flask</span> <span>import</span> <span>Request</span>
<span>SECRET</span> <span>=</span> <span>bytes</span><span>(</span><span>os</span><span>.</span><span>environ</span><span>[</span><span>"</span><span>SECRET</span><span>"</span><span>],</span> <span>"</span><span>utf-8</span><span>"</span><span>)</span>
<span>def</span> <span>verify_signature</span><span>(</span><span>request</span><span>:</span> <span>Request</span><span>)</span> <span>-></span> <span>bool</span><span>:</span>
<span>"""</span><span> Validates the github webhook secret. Will return false if secret not provided </span><span>"""</span>
<span>expected_signature</span> <span>=</span> <span>hmac</span><span>.</span><span>new</span><span>(</span>
<span>key</span><span>=</span><span>SECRET</span><span>,</span> <span>msg</span><span>=</span><span>request</span><span>.</span><span>data</span><span>,</span> <span>digestmod</span><span>=</span><span>hashlib</span><span>.</span><span>sha256</span>
<span>).</span><span>hexdigest</span><span>()</span>
<span>incoming_signature</span> <span>=</span> <span>request</span><span>.</span><span>headers</span><span>.</span><span>get</span><span>(</span><span>"</span><span>X-Hub-Signature-256</span><span>"</span><span>,</span> <span>""</span><span>).</span><span>removeprefix</span><span>(</span>
<span>"</span><span>sha256=</span><span>"</span>
<span>)</span>
<span>return</span> <span>hmac</span><span>.</span><span>compare_digest</span><span>(</span><span>incoming_signature</span><span>,</span> <span>expected_signature</span><span>)</span>
<span>"""</span><span> Verify the GitHub webhook secret </span><span>"""</span>
<span>import</span> <span>os</span>
<span>import</span> <span>hmac</span>
<span>import</span> <span>hashlib</span>

<span>from</span> <span>flask</span> <span>import</span> <span>Request</span>

<span>SECRET</span> <span>=</span> <span>bytes</span><span>(</span><span>os</span><span>.</span><span>environ</span><span>[</span><span>"</span><span>SECRET</span><span>"</span><span>],</span> <span>"</span><span>utf-8</span><span>"</span><span>)</span>


<span>def</span> <span>verify_signature</span><span>(</span><span>request</span><span>:</span> <span>Request</span><span>)</span> <span>-></span> <span>bool</span><span>:</span>
    <span>"""</span><span> Validates the github webhook secret. Will return false if secret not provided </span><span>"""</span>
    <span>expected_signature</span> <span>=</span> <span>hmac</span><span>.</span><span>new</span><span>(</span>
        <span>key</span><span>=</span><span>SECRET</span><span>,</span> <span>msg</span><span>=</span><span>request</span><span>.</span><span>data</span><span>,</span> <span>digestmod</span><span>=</span><span>hashlib</span><span>.</span><span>sha256</span>
    <span>).</span><span>hexdigest</span><span>()</span>
    <span>incoming_signature</span> <span>=</span> <span>request</span><span>.</span><span>headers</span><span>.</span><span>get</span><span>(</span><span>"</span><span>X-Hub-Signature-256</span><span>"</span><span>,</span> <span>""</span><span>).</span><span>removeprefix</span><span>(</span>
        <span>"</span><span>sha256=</span><span>"</span>
    <span>)</span>
    <span>return</span> <span>hmac</span><span>.</span><span>compare_digest</span><span>(</span><span>incoming_signature</span><span>,</span> <span>expected_signature</span><span>)</span>
""" Verify the GitHub webhook secret """ import os import hmac import hashlib from flask import Request SECRET = bytes(os.environ["SECRET"], "utf-8") def verify_signature(request: Request) -> bool: """ Validates the github webhook secret. Will return false if secret not provided """ expected_signature = hmac.new( key=SECRET, msg=request.data, digestmod=hashlib.sha256 ).hexdigest() incoming_signature = request.headers.get("X-Hub-Signature-256", "").removeprefix( "sha256=" ) return hmac.compare_digest(incoming_signature, expected_signature)

Enter fullscreen mode Exit fullscreen mode

A couple of things to note here: the new Python 3.9 .removeprefix() method for strings, and using hmac.compare_digest() to compare the two digests. This method is equivalent to just doing == but it introduces some random timing to prevent time analysis, which makes it a bit more secure (though in our case that is unlikely to be a major consideration, it’s still best-practice)

Main function

So finally, the main function that we will deploy looks like this (full contents of app/main.py):

<span>"""</span><span> Listens to webhooks from GitHub </span><span>"""</span>
<span>import</span> <span>structlog</span> <span># type: ignore </span><span>from</span> <span>flask</span> <span>import</span> <span>Request</span><span>,</span> <span>abort</span>
<span>from</span> <span>pydantic</span> <span>import</span> <span>ValidationError</span>
<span>from</span> <span>utils.verify</span> <span>import</span> <span>verify_signature</span>
<span>from</span> <span>utils.models</span> <span>import</span> <span>GitHubHookFork</span>
<span>logger</span> <span>=</span> <span>structlog</span><span>.</span><span>get_logger</span><span>()</span>
<span>OUR_REPO</span> <span>=</span> <span>"</span><span>meseta/lgtm</span><span>"</span>
<span>def</span> <span>github_webhook_listener</span><span>(</span><span>request</span><span>:</span> <span>Request</span><span>):</span>
<span>"""</span><span> A listener for github webhooks </span><span>"""</span>
<span># verify </span> <span>if</span> <span>not</span> <span>verify_signature</span><span>(</span><span>request</span><span>):</span>
<span>return</span> <span>abort</span><span>(</span><span>403</span><span>,</span> <span>"</span><span>Invalid signature</span><span>"</span><span>)</span>
<span># decode </span> <span>try</span><span>:</span>
<span>hook_fork</span> <span>=</span> <span>GitHubHookFork</span><span>.</span><span>parse_raw</span><span>(</span><span>request</span><span>.</span><span>data</span><span>)</span>
<span>except</span> <span>ValidationError</span> <span>as</span> <span>err</span><span>:</span>
<span>logger</span><span>.</span><span>err</span><span>(</span><span>"</span><span>Validation error</span><span>"</span><span>,</span> <span>err</span><span>=</span><span>err</span><span>)</span>
<span>return</span> <span>abort</span><span>(</span><span>400</span><span>,</span> <span>"</span><span>Validation error</span><span>"</span><span>)</span>
<span># output </span> <span>if</span> <span>hook_fork</span><span>.</span><span>repository</span><span>.</span><span>full_name</span> <span>==</span> <span>OUR_REPO</span><span>:</span>
<span>logger</span><span>.</span><span>info</span><span>(</span><span>"</span><span>Got fork</span><span>"</span><span>,</span> <span>data</span><span>=</span><span>hook_fork</span><span>.</span><span>dict</span><span>())</span>
<span>return</span> <span>"</span><span>OK</span><span>"</span>
<span>"""</span><span> Listens to webhooks from GitHub </span><span>"""</span>

<span>import</span> <span>structlog</span>  <span># type: ignore </span><span>from</span> <span>flask</span> <span>import</span> <span>Request</span><span>,</span> <span>abort</span>
<span>from</span> <span>pydantic</span> <span>import</span> <span>ValidationError</span>

<span>from</span> <span>utils.verify</span> <span>import</span> <span>verify_signature</span>
<span>from</span> <span>utils.models</span> <span>import</span> <span>GitHubHookFork</span>

<span>logger</span> <span>=</span> <span>structlog</span><span>.</span><span>get_logger</span><span>()</span>

<span>OUR_REPO</span> <span>=</span> <span>"</span><span>meseta/lgtm</span><span>"</span>


<span>def</span> <span>github_webhook_listener</span><span>(</span><span>request</span><span>:</span> <span>Request</span><span>):</span>
    <span>"""</span><span> A listener for github webhooks </span><span>"""</span>

    <span># verify </span>    <span>if</span> <span>not</span> <span>verify_signature</span><span>(</span><span>request</span><span>):</span>
        <span>return</span> <span>abort</span><span>(</span><span>403</span><span>,</span> <span>"</span><span>Invalid signature</span><span>"</span><span>)</span>

    <span># decode </span>    <span>try</span><span>:</span>
        <span>hook_fork</span> <span>=</span> <span>GitHubHookFork</span><span>.</span><span>parse_raw</span><span>(</span><span>request</span><span>.</span><span>data</span><span>)</span>
    <span>except</span> <span>ValidationError</span> <span>as</span> <span>err</span><span>:</span>
        <span>logger</span><span>.</span><span>err</span><span>(</span><span>"</span><span>Validation error</span><span>"</span><span>,</span> <span>err</span><span>=</span><span>err</span><span>)</span>
        <span>return</span> <span>abort</span><span>(</span><span>400</span><span>,</span> <span>"</span><span>Validation error</span><span>"</span><span>)</span>

    <span># output </span>    <span>if</span> <span>hook_fork</span><span>.</span><span>repository</span><span>.</span><span>full_name</span> <span>==</span> <span>OUR_REPO</span><span>:</span>
        <span>logger</span><span>.</span><span>info</span><span>(</span><span>"</span><span>Got fork</span><span>"</span><span>,</span> <span>data</span><span>=</span><span>hook_fork</span><span>.</span><span>dict</span><span>())</span>

    <span>return</span> <span>"</span><span>OK</span><span>"</span>
""" Listens to webhooks from GitHub """ import structlog # type: ignore from flask import Request, abort from pydantic import ValidationError from utils.verify import verify_signature from utils.models import GitHubHookFork logger = structlog.get_logger() OUR_REPO = "meseta/lgtm" def github_webhook_listener(request: Request): """ A listener for github webhooks """ # verify if not verify_signature(request): return abort(403, "Invalid signature") # decode try: hook_fork = GitHubHookFork.parse_raw(request.data) except ValidationError as err: logger.err("Validation error", err=err) return abort(400, "Validation error") # output if hook_fork.repository.full_name == OUR_REPO: logger.info("Got fork", data=hook_fork.dict()) return "OK"

Enter fullscreen mode Exit fullscreen mode

Note that at this point, I’m not actually doing anything with the data, just decoding it and logging it. Next sprint’s task will be to actually take this data and use it to create accounts and whatever.

Tests

To ensure we have full test coverage, I wrote a couple of tests, as well as downloaded the payload and hash for a valid hook, and fabricated an invalid one. The tests/conftest.py file now looks like this, with fixtures to provide the two test payloads as well as the test client:

<span>"""</span><span> Setup for tests </span><span>"""</span>
<span>import</span> <span>os</span>
<span>import</span> <span>json</span>
<span>import</span> <span>pytest</span>
<span>from</span> <span>functions_framework</span> <span>import</span> <span>create_app</span> <span># type: ignore </span>
<span>TEST_FILES</span> <span>=</span> <span>os</span><span>.</span><span>path</span><span>.</span><span>join</span><span>(</span>
<span>os</span><span>.</span><span>path</span><span>.</span><span>dirname</span><span>(</span><span>os</span><span>.</span><span>path</span><span>.</span><span>realpath</span><span>(</span><span>__file__</span><span>)),</span>
<span>"</span><span>test_files</span><span>"</span><span>,</span>
<span>)</span>
<span>class</span> <span>Payload</span><span>:</span> <span># pylint: disable=too-few-public-methods </span> <span>"""</span><span> Container for holding header/payload pairs during testing</span><span>"""</span>
<span>def</span> <span>__init__</span><span>(</span><span>self</span><span>,</span> <span>header_path</span><span>,</span> <span>payload_path</span><span>):</span>
<span>with</span> <span>open</span><span>(</span><span>os</span><span>.</span><span>path</span><span>.</span><span>join</span><span>(</span><span>TEST_FILES</span><span>,</span> <span>header_path</span><span>))</span> <span>as</span> <span>fp</span><span>:</span>
<span>self</span><span>.</span><span>headers</span> <span>=</span> <span>json</span><span>.</span><span>load</span><span>(</span><span>fp</span><span>)</span>
<span>with</span> <span>open</span><span>(</span><span>os</span><span>.</span><span>path</span><span>.</span><span>join</span><span>(</span><span>TEST_FILES</span><span>,</span> <span>payload_path</span><span>),</span> <span>"</span><span>rb</span><span>"</span><span>)</span> <span>as</span> <span>fp</span><span>:</span>
<span>self</span><span>.</span><span>payload</span> <span>=</span> <span>fp</span><span>.</span><span>read</span><span>()</span>
<span>@pytest.fixture</span><span>()</span>
<span>def</span> <span>good_fork</span><span>():</span>
<span>"""</span><span> A payload containing (raw) data, this is recorded from GitHub </span><span>"""</span>
<span>return</span> <span>Payload</span><span>(</span><span>"</span><span>good_fork_headers.json</span><span>"</span><span>,</span> <span>"</span><span>good_fork.bin</span><span>"</span><span>)</span>
<span>@pytest.fixture</span><span>()</span>
<span>def</span> <span>bad_fork</span><span>():</span>
<span>"""</span><span>A payload containing (raw) data, that has been edited to be missing stuff but paired wiht a valid signature</span><span>"""</span>
<span>return</span> <span>Payload</span><span>(</span><span>"</span><span>bad_fork_headers.json</span><span>"</span><span>,</span> <span>"</span><span>bad_fork.bin</span><span>"</span><span>)</span>
<span>@pytest.fixture</span><span>(</span><span>scope</span><span>=</span><span>"</span><span>package</span><span>"</span><span>)</span>
<span>def</span> <span>client</span><span>():</span>
<span>"""</span><span> Test client </span><span>"""</span>
<span>return</span> <span>create_app</span><span>(</span>
<span>"</span><span>github_webhook_listener</span><span>"</span><span>,</span> <span>os</span><span>.</span><span>environ</span><span>[</span><span>"</span><span>FUNCTION_SOURCE</span><span>"</span><span>]</span>
<span>).</span><span>test_client</span><span>()</span>
<span>"""</span><span> Setup for tests </span><span>"""</span>

<span>import</span> <span>os</span>
<span>import</span> <span>json</span>
<span>import</span> <span>pytest</span>
<span>from</span> <span>functions_framework</span> <span>import</span> <span>create_app</span>  <span># type: ignore </span>
<span>TEST_FILES</span> <span>=</span> <span>os</span><span>.</span><span>path</span><span>.</span><span>join</span><span>(</span>
    <span>os</span><span>.</span><span>path</span><span>.</span><span>dirname</span><span>(</span><span>os</span><span>.</span><span>path</span><span>.</span><span>realpath</span><span>(</span><span>__file__</span><span>)),</span>
    <span>"</span><span>test_files</span><span>"</span><span>,</span>
<span>)</span>


<span>class</span> <span>Payload</span><span>:</span>  <span># pylint: disable=too-few-public-methods </span>    <span>"""</span><span> Container for holding header/payload pairs during testing</span><span>"""</span>

    <span>def</span> <span>__init__</span><span>(</span><span>self</span><span>,</span> <span>header_path</span><span>,</span> <span>payload_path</span><span>):</span>
        <span>with</span> <span>open</span><span>(</span><span>os</span><span>.</span><span>path</span><span>.</span><span>join</span><span>(</span><span>TEST_FILES</span><span>,</span> <span>header_path</span><span>))</span> <span>as</span> <span>fp</span><span>:</span>
            <span>self</span><span>.</span><span>headers</span> <span>=</span> <span>json</span><span>.</span><span>load</span><span>(</span><span>fp</span><span>)</span>
        <span>with</span> <span>open</span><span>(</span><span>os</span><span>.</span><span>path</span><span>.</span><span>join</span><span>(</span><span>TEST_FILES</span><span>,</span> <span>payload_path</span><span>),</span> <span>"</span><span>rb</span><span>"</span><span>)</span> <span>as</span> <span>fp</span><span>:</span>
            <span>self</span><span>.</span><span>payload</span> <span>=</span> <span>fp</span><span>.</span><span>read</span><span>()</span>


<span>@pytest.fixture</span><span>()</span>
<span>def</span> <span>good_fork</span><span>():</span>
    <span>"""</span><span> A payload containing (raw) data, this is recorded from GitHub </span><span>"""</span>
    <span>return</span> <span>Payload</span><span>(</span><span>"</span><span>good_fork_headers.json</span><span>"</span><span>,</span> <span>"</span><span>good_fork.bin</span><span>"</span><span>)</span>


<span>@pytest.fixture</span><span>()</span>
<span>def</span> <span>bad_fork</span><span>():</span>
    <span>"""</span><span>A payload containing (raw) data, that has been edited to be missing stuff but paired wiht a valid signature</span><span>"""</span>
    <span>return</span> <span>Payload</span><span>(</span><span>"</span><span>bad_fork_headers.json</span><span>"</span><span>,</span> <span>"</span><span>bad_fork.bin</span><span>"</span><span>)</span>


<span>@pytest.fixture</span><span>(</span><span>scope</span><span>=</span><span>"</span><span>package</span><span>"</span><span>)</span>
<span>def</span> <span>client</span><span>():</span>
    <span>"""</span><span> Test client </span><span>"""</span>
    <span>return</span> <span>create_app</span><span>(</span>
        <span>"</span><span>github_webhook_listener</span><span>"</span><span>,</span> <span>os</span><span>.</span><span>environ</span><span>[</span><span>"</span><span>FUNCTION_SOURCE</span><span>"</span><span>]</span>
    <span>).</span><span>test_client</span><span>()</span>
""" Setup for tests """ import os import json import pytest from functions_framework import create_app # type: ignore TEST_FILES = os.path.join( os.path.dirname(os.path.realpath(__file__)), "test_files", ) class Payload: # pylint: disable=too-few-public-methods """ Container for holding header/payload pairs during testing""" def __init__(self, header_path, payload_path): with open(os.path.join(TEST_FILES, header_path)) as fp: self.headers = json.load(fp) with open(os.path.join(TEST_FILES, payload_path), "rb") as fp: self.payload = fp.read() @pytest.fixture() def good_fork(): """ A payload containing (raw) data, this is recorded from GitHub """ return Payload("good_fork_headers.json", "good_fork.bin") @pytest.fixture() def bad_fork(): """A payload containing (raw) data, that has been edited to be missing stuff but paired wiht a valid signature""" return Payload("bad_fork_headers.json", "bad_fork.bin") @pytest.fixture(scope="package") def client(): """ Test client """ return create_app( "github_webhook_listener", os.environ["FUNCTION_SOURCE"] ).test_client()

Enter fullscreen mode Exit fullscreen mode

And to test our model (contents of tests/test_model.py), we simply ask it to parse the good and the bad payloads to see if it’ll work or raise the appropriate exceptions:

<span>"""</span><span> Tests for pydantic models </span><span>"""</span>
<span>import</span> <span>pytest</span>
<span>from</span> <span>pydantic</span> <span>import</span> <span>ValidationError</span>
<span>from</span> <span>app.utils.models</span> <span>import</span> <span>GitHubHookFork</span>
<span># pylint: disable=redefined-outer-name </span><span>def</span> <span>test_model</span><span>(</span><span>good_fork</span><span>):</span>
<span>"""</span><span> Test that our validator works </span><span>"""</span>
<span>hook_fork</span> <span>=</span> <span>GitHubHookFork</span><span>.</span><span>parse_raw</span><span>(</span><span>good_fork</span><span>.</span><span>payload</span><span>)</span>
<span>assert</span> <span>hook_fork</span><span>.</span><span>forkee</span><span>.</span><span>owner</span><span>.</span><span>login</span>
<span>def</span> <span>test_model_invalid</span><span>(</span><span>bad_fork</span><span>):</span>
<span>"""</span><span> Test that our validator fails correctly </span><span>"""</span>
<span>with</span> <span>pytest</span><span>.</span><span>raises</span><span>(</span><span>ValidationError</span><span>):</span>
<span>GitHubHookFork</span><span>.</span><span>parse_raw</span><span>(</span><span>bad_fork</span><span>.</span><span>payload</span><span>)</span>
<span>"""</span><span> Tests for pydantic models </span><span>"""</span>

<span>import</span> <span>pytest</span>
<span>from</span> <span>pydantic</span> <span>import</span> <span>ValidationError</span>
<span>from</span> <span>app.utils.models</span> <span>import</span> <span>GitHubHookFork</span>

<span># pylint: disable=redefined-outer-name </span><span>def</span> <span>test_model</span><span>(</span><span>good_fork</span><span>):</span>
    <span>"""</span><span> Test that our validator works </span><span>"""</span>

    <span>hook_fork</span> <span>=</span> <span>GitHubHookFork</span><span>.</span><span>parse_raw</span><span>(</span><span>good_fork</span><span>.</span><span>payload</span><span>)</span>
    <span>assert</span> <span>hook_fork</span><span>.</span><span>forkee</span><span>.</span><span>owner</span><span>.</span><span>login</span>


<span>def</span> <span>test_model_invalid</span><span>(</span><span>bad_fork</span><span>):</span>
    <span>"""</span><span> Test that our validator fails correctly </span><span>"""</span>

    <span>with</span> <span>pytest</span><span>.</span><span>raises</span><span>(</span><span>ValidationError</span><span>):</span>
        <span>GitHubHookFork</span><span>.</span><span>parse_raw</span><span>(</span><span>bad_fork</span><span>.</span><span>payload</span><span>)</span>
""" Tests for pydantic models """ import pytest from pydantic import ValidationError from app.utils.models import GitHubHookFork # pylint: disable=redefined-outer-name def test_model(good_fork): """ Test that our validator works """ hook_fork = GitHubHookFork.parse_raw(good_fork.payload) assert hook_fork.forkee.owner.login def test_model_invalid(bad_fork): """ Test that our validator fails correctly """ with pytest.raises(ValidationError): GitHubHookFork.parse_raw(bad_fork.payload)

Enter fullscreen mode Exit fullscreen mode

And to test our main function, there’s four tests that test various combinations of valid/invalid/missing signatures, valid/invalid payloads (contents of tests/test_main.py)

<span>"""</span><span> Tests for main.py </span><span>"""</span>
<span># pylint: disable=redefined-outer-name </span><span>def</span> <span>test_good_fork</span><span>(</span><span>client</span><span>,</span> <span>good_fork</span><span>):</span>
<span>"""</span><span> For a good fork that</span><span>'</span><span>s working fine </span><span>"""</span>
<span>res</span> <span>=</span> <span>client</span><span>.</span><span>post</span><span>(</span><span>"</span><span>/</span><span>"</span><span>,</span> <span>headers</span><span>=</span><span>good_fork</span><span>.</span><span>headers</span><span>,</span> <span>data</span><span>=</span><span>good_fork</span><span>.</span><span>payload</span><span>)</span>
<span>assert</span> <span>res</span><span>.</span><span>status_code</span> <span>==</span> <span>200</span>
<span>def</span> <span>test_model_validation_fail</span><span>(</span><span>client</span><span>,</span> <span>bad_fork</span><span>):</span>
<span>"""</span><span> Test model validation failure with bad payload but correct signature for it</span><span>"""</span>
<span>res</span> <span>=</span> <span>client</span><span>.</span><span>post</span><span>(</span><span>"</span><span>/</span><span>"</span><span>,</span> <span>headers</span><span>=</span><span>bad_fork</span><span>.</span><span>headers</span><span>,</span> <span>data</span><span>=</span><span>bad_fork</span><span>.</span><span>payload</span><span>)</span>
<span>assert</span> <span>res</span><span>.</span><span>status_code</span> <span>==</span> <span>400</span>
<span>def</span> <span>test_signature_fail</span><span>(</span><span>client</span><span>,</span> <span>good_fork</span><span>,</span> <span>bad_fork</span><span>):</span>
<span>"""</span><span> Test signature validation failure with good payload but incorrect signature for it</span><span>"""</span>
<span>res</span> <span>=</span> <span>client</span><span>.</span><span>post</span><span>(</span><span>"</span><span>/</span><span>"</span><span>,</span> <span>headers</span><span>=</span><span>bad_fork</span><span>.</span><span>headers</span><span>,</span> <span>data</span><span>=</span><span>good_fork</span><span>.</span><span>payload</span><span>)</span>
<span>assert</span> <span>res</span><span>.</span><span>status_code</span> <span>==</span> <span>403</span>
<span>def</span> <span>test_no_signature</span><span>(</span><span>client</span><span>,</span> <span>good_fork</span><span>):</span>
<span>"""</span><span> When signatures are not supplied </span><span>"""</span>
<span>res</span> <span>=</span> <span>client</span><span>.</span><span>post</span><span>(</span>
<span>"</span><span>/</span><span>"</span><span>,</span> <span>headers</span><span>=</span><span>{</span><span>"</span><span>Content-Type</span><span>"</span><span>:</span> <span>"</span><span>application/json</span><span>"</span><span>},</span> <span>data</span><span>=</span><span>good_fork</span><span>.</span><span>payload</span>
<span>)</span>
<span>assert</span> <span>res</span><span>.</span><span>status_code</span> <span>==</span> <span>403</span>
<span>"""</span><span> Tests for main.py </span><span>"""</span>

<span># pylint: disable=redefined-outer-name </span><span>def</span> <span>test_good_fork</span><span>(</span><span>client</span><span>,</span> <span>good_fork</span><span>):</span>
    <span>"""</span><span> For a good fork that</span><span>'</span><span>s working fine </span><span>"""</span>

    <span>res</span> <span>=</span> <span>client</span><span>.</span><span>post</span><span>(</span><span>"</span><span>/</span><span>"</span><span>,</span> <span>headers</span><span>=</span><span>good_fork</span><span>.</span><span>headers</span><span>,</span> <span>data</span><span>=</span><span>good_fork</span><span>.</span><span>payload</span><span>)</span>
    <span>assert</span> <span>res</span><span>.</span><span>status_code</span> <span>==</span> <span>200</span>


<span>def</span> <span>test_model_validation_fail</span><span>(</span><span>client</span><span>,</span> <span>bad_fork</span><span>):</span>
    <span>"""</span><span> Test model validation failure with bad payload but correct signature for it</span><span>"""</span>

    <span>res</span> <span>=</span> <span>client</span><span>.</span><span>post</span><span>(</span><span>"</span><span>/</span><span>"</span><span>,</span> <span>headers</span><span>=</span><span>bad_fork</span><span>.</span><span>headers</span><span>,</span> <span>data</span><span>=</span><span>bad_fork</span><span>.</span><span>payload</span><span>)</span>
    <span>assert</span> <span>res</span><span>.</span><span>status_code</span> <span>==</span> <span>400</span>


<span>def</span> <span>test_signature_fail</span><span>(</span><span>client</span><span>,</span> <span>good_fork</span><span>,</span> <span>bad_fork</span><span>):</span>
    <span>"""</span><span> Test signature validation failure with good payload but incorrect signature for it</span><span>"""</span>

    <span>res</span> <span>=</span> <span>client</span><span>.</span><span>post</span><span>(</span><span>"</span><span>/</span><span>"</span><span>,</span> <span>headers</span><span>=</span><span>bad_fork</span><span>.</span><span>headers</span><span>,</span> <span>data</span><span>=</span><span>good_fork</span><span>.</span><span>payload</span><span>)</span>
    <span>assert</span> <span>res</span><span>.</span><span>status_code</span> <span>==</span> <span>403</span>


<span>def</span> <span>test_no_signature</span><span>(</span><span>client</span><span>,</span> <span>good_fork</span><span>):</span>
    <span>"""</span><span> When signatures are not supplied </span><span>"""</span>

    <span>res</span> <span>=</span> <span>client</span><span>.</span><span>post</span><span>(</span>
        <span>"</span><span>/</span><span>"</span><span>,</span> <span>headers</span><span>=</span><span>{</span><span>"</span><span>Content-Type</span><span>"</span><span>:</span> <span>"</span><span>application/json</span><span>"</span><span>},</span> <span>data</span><span>=</span><span>good_fork</span><span>.</span><span>payload</span>
    <span>)</span>
    <span>assert</span> <span>res</span><span>.</span><span>status_code</span> <span>==</span> <span>403</span>
""" Tests for main.py """ # pylint: disable=redefined-outer-name def test_good_fork(client, good_fork): """ For a good fork that's working fine """ res = client.post("/", headers=good_fork.headers, data=good_fork.payload) assert res.status_code == 200 def test_model_validation_fail(client, bad_fork): """ Test model validation failure with bad payload but correct signature for it""" res = client.post("/", headers=bad_fork.headers, data=bad_fork.payload) assert res.status_code == 400 def test_signature_fail(client, good_fork, bad_fork): """ Test signature validation failure with good payload but incorrect signature for it""" res = client.post("/", headers=bad_fork.headers, data=good_fork.payload) assert res.status_code == 403 def test_no_signature(client, good_fork): """ When signatures are not supplied """ res = client.post( "/", headers={"Content-Type": "application/json"}, data=good_fork.payload ) assert res.status_code == 403

Enter fullscreen mode Exit fullscreen mode

Our final test results show all tests pass, with 100% code coverage, which isn’t definitive that our code is bug free, but gives us a little confidence, and helps us make sure there’s no regressions when we edit the code later

Running a real test

To run a real test, I fire up that ngrok endpoint I set up previously (by running ngrok http 5000), started up the test server using pipenv run serve github_webhook_listener, updating the GitHub Webhook URL to the new temporary ngrok path, and telling GitHub to re-send the previous webhook.

The server logs show that a webhook is received, and you can read the decoded variables that are now in the hook_fork object!

For those interested, all of this is running inside VSCode’s Terminal pane, opened into WSL, and inside a tmux session. It’s a nice way to run code while developing.


This represents all that is needed for a basic GitHub webhook receiving serverless function, that is built in a reasonably robust way. Next we need to upload it, but I will set up the CI to do this instead of me having to upload it every time.

LGTM (40 Part Series)

1 LGTM Devlog 0: Teaching Git through playing a game
2 LGTM Devlog 1: New Logo, and jumping the gun by buying a domain
36 more parts…
3 LGTM Devlog 2: Design Ideas for the game I
4 LGTM Devlog 3: Design Ideas for the game 2
5 LGTM Devlog 4: Technical Decisions – GitHub API and Python
6 LGTM Devlog 5: Technical Decisions – Serverless Architecture
7 LGTM Devlog 6: Sprint 1 plans
8 LGTM Devlog 7: Creating a Firebase project
9 LGTM Devlog 8: Starting a git repo for an open source project from scratch
10 LGTM Devlog 9: Python Google Cloud Functions with Unit Tests and Linting
11 LGTM Devlog 10: Capturing the GitHub webhook for fork requests
12 LGTM Devlog 11: Writing the Serverless Function for receiving GitHub webhooks with Pydantic validation
13 LGTM Devlog 12: CI/CD with GitHub Actions to run Unit Tests and deploy Firebase Functions
14 LGTM Devlog 13: GitHub Branch Protection and Security
15 LGTM Devlog 14: Sprint 1 Retrospective
16 LGTM Devlog 15: Sprint 2 plans
17 LGTM Devlog 16: A serverless data base access rule conundrum
18 LGTM Devlog 17: Website and GitHub OAuth
19 LGTM Devlog 18: Python Serverless functions using GitHub API to validate users
20 LGTM Devlog 19: Game data/quest storage
21 LGTM Devlog 20: Python Abstract Base Class-based data/quest storage
22 LGTM Devlog 21: Deploying Pub/Sub-triggered Python Google Cloud Functions
23 LGTM Devlog 22: Modularization
24 LGTM Devlog 23: Sprint 2 Retrospective
25 LGTM Devlog 24: Sprint 3 plans
26 LGTM Devlog 25: Some cleanup
27 LGTM Devlog 26: Python Graphlib DAGs for Quest Stages
28 LGTM Devlog 27: Branching quests
29 LGTM Devlog 28: Game event loop using Google Cloud Scheduler and PubSub
30 LGTM Devlog 29: ORM for Firestore and __init__subclass__ dunders and metaclasses
31 LGTM Devlog 30: Sprint 3 Retrospective
32 LGTM Devlog 31: Sprint 4 Plan
33 LGTM Devlog 32: Secrets Management to avoid storing API keys in services
34 LGTM Devlog 33: Using PyGithub to post GitHub issues and comments
35 LGTM Devlog 34: Characters Posting on GitHub Issues
36 LGTM Devlog 35: Responding to the player’s answers on GitHub Issues Comments
37 LGTM Devlog 36: Character profiles! On GitHub
38 LGTM Devlog 37: Sprint 4 Retrospective
39 LGTM Devlog 38: Sprint 5 Plan
40 LGTM Devlog 39: Planning the story

原文链接:LGTM Devlog 11: Writing the Serverless Function for receiving GitHub webhooks with Pydantic validation

© 版权声明
THE END
喜欢就支持一下吧
点赞5 分享
Happiness will never miss any people, sooner or later it will find you.
幸福不会遗漏任何人,迟早有一天它会找到你
评论 抢沙发

请登录后发表评论

    暂无评论内容