Creating a Datalake in AWS for NBA analytics!

Hey there thanks for stopping by! Today, I will be teaching you how to automatically spin up an AWS S3 bucket, create a Glue database and fetch NBA data to store in the bucket. From there we will utilize AWS Athena to query the data. Stick around to learn more on AWS S3, Glue, Athena and Python!

Purpose

The purpose of this article is to walk you through how to leverage AWS tools and technologies to fetch, store and visualize data. The technologies that will be used in this article include: GitHub, AWS (S3, Athena, Glue) and Python.

Resources

Discord
YouTube Video from Alicia
My GitHub Repo

Prerequisites

VS Code (or your favorite text editor)
AWS Free Tier Account
GitHub Account
SportsData.io Account and API key
Knowledge of Git, Linux commands, APIs and AWS

Architecture

Setup

1. Clone the repo

```
git clone https://github.com/asciikeyboard/nba-datalake.git
cd nba-datalake
```

Enter fullscreen mode Exit fullscreen mode

2. Log into the AWS console and launch CloudShell

3. Create the Python file

• In the CLI, type nano setup_nba_data_lake.py
• Copy and paste the contents from the setup_nba_data_lake.py file in the src folder in GitHub into the CloudShell terminal
• Press ^X to exit, press Y to save the file, press enter to confirm the file name

4. Create the environment variables file

• In the CLI, type nano .env
• Paste the following info into the file and update with your API key

SPORTS_DATA_API_KEY=your_sportsdata_api_key
NBA_ENDPOINT=https://api.sportsdata.io/v3/nba/scores/json/Players

Enter fullscreen mode Exit fullscreen mode

• Press ^X to exit, press Y to save the file, press enter to confirm the file name

5. Install dotenv in CloudShell and run the script

• Type pip install python-dotenv and hit enter
• Once dotenv is installed, run the python script python3 setup_nba_data_lake.py. You should see the following output if successful:

6. Verify resources are in AWS
In IT there is a phrase that goes like this “trust but verify”. That’s what we are going to do now.

• Search for S3 in the AWS console and check to see if your bucket is there
• Next, search for Athena in the AWS console and run a query against the newly populated data:

SELECT FirstName, LastName, Position, Team
FROM nba_players
WHERE Position = 'PG';

Enter fullscreen mode Exit fullscreen mode

The result should look like this:

Recap

Now, you have successfully created a Python script that automagically creates an S3 bucket and Glue database with NBA data. You then learned how to leverage Amazon Athena to visualize that data. You are well on your way to becoming a cloudy!

原文链接：Creating a Datalake in AWS for NBA analytics!