How to Check Internal and External links on a Webpage using Python

Python is one of the most popular programming languages for extracting, processing, and analyzing data. The inbuilt and third-party libraries of Python make it very easy for a developer to get specific data from a web page and make results around those data sets.
In this article, I have covered a simple Python script that can extract links from a given url of a web page and create a CSV file containing all the links present on that web page with extra information telling whether the link is external or internal.

Prerequisite

As it is a python article with the program, so it goes without saying that you need to have basic knowledge of Python and Python installed on your system to test the program for yourself.
If you are on a new system, you can easily install the latest version of Python with this quick download link.

To make the program, I will use 4 Python libraries, among which two libraries are third-party libraries, and the other two are built-in.

Libraries

1. requests:
requests is the popular python HTTP library. We will use this library to make an HTTP request for the url which links we want to check.
As requests is a third-party library, we need to install it for our Python environment using the pip command.
pip install requests
2. Beautiful soup:
Beautiful soup is a third-party Python library that can extract data from HTML and XML files. Generally, a web page is an HTML document, and we can use the Python beautiful soup to extract links from that web page.

Use the following command to install beautiful soup
pip install beautifulsoup4
3. csv
csv modules come with Python, and we can write, read and append between .csv files using this module.

4. datetime
datetime is also an inbuilt Python module that can deal with date and time.

Program

Now let’s use all these 4 Python modules and write a Program that can tell all the internal and external links of a web page and export that data into a .csv file.

I have divided this program into three functions to make it modular.

Function 1: requestMaker(url)

The requestMake(url) function accepts the url as a string and sends a get request to the url using the .get() method.
After making the request, inside the requestMaker() function, I collected the response web page HTML content and the url using the .text and .url properties.
And called the parseLinks(pageHtml, pageUrl) function.


<span>#to make the HTTP request to the give url </span><span>def</span> <span>requestMaker</span><span>(</span><span>url</span><span>):</span>
    <span>try</span><span>:</span>
        <span>#make the get request to the url </span>        <span>response</span> <span>=</span> <span>requests</span><span>.</span><span>get</span><span>(</span><span>url</span><span>)</span>
        <span>#if the request is successful </span>        <span>if</span> <span>response</span><span>.</span><span>status_code</span> <span>in</span> <span>range</span><span>(</span><span>200</span><span>,</span> <span>300</span><span>):</span>
            <span>#extract the page html content for parsing the links </span>            <span>pageHtml</span> <span>=</span> <span>response</span><span>.</span><span>text</span>
            <span>pageUrl</span> <span>=</span> <span>response</span><span>.</span><span>url</span>
            <span>#call the parseLink function </span>            <span>parseLinks</span><span>(</span><span>pageHtml</span><span>,</span> <span>pageUrl</span><span>)</span>
        <span>else</span><span>:</span>
            <span>print</span><span>(</span><span>"</span><span>Sorry Could not fetch the result status code {response.status_code}!</span><span>"</span><span>)</span>
    <span>except</span><span>:</span>
        <span>print</span><span>(</span><span>f</span><span>"</span><span>Could Not Connect to url </span><span>{</span><span>url</span><span>}</span><span>"</span><span>)</span>
<span>#to make the HTTP request to the give url </span><span>def</span> <span>requestMaker</span><span>(</span><span>url</span><span>):</span>
    <span>try</span><span>:</span>
        <span>#make the get request to the url </span>        <span>response</span> <span>=</span> <span>requests</span><span>.</span><span>get</span><span>(</span><span>url</span><span>)</span>

        <span>#if the request is successful </span>        <span>if</span> <span>response</span><span>.</span><span>status_code</span> <span>in</span> <span>range</span><span>(</span><span>200</span><span>,</span> <span>300</span><span>):</span>
            <span>#extract the page html content for parsing the links </span>            <span>pageHtml</span> <span>=</span> <span>response</span><span>.</span><span>text</span>
            <span>pageUrl</span> <span>=</span> <span>response</span><span>.</span><span>url</span>

            <span>#call the parseLink function </span>            <span>parseLinks</span><span>(</span><span>pageHtml</span><span>,</span> <span>pageUrl</span><span>)</span>

        <span>else</span><span>:</span>
            <span>print</span><span>(</span><span>"</span><span>Sorry Could not fetch the result status code {response.status_code}!</span><span>"</span><span>)</span>

    <span>except</span><span>:</span>
        <span>print</span><span>(</span><span>f</span><span>"</span><span>Could Not Connect to url </span><span>{</span><span>url</span><span>}</span><span>"</span><span>)</span>


#to make the HTTP request to the give url def requestMaker(url):
    try:
        #make the get request to the url         response = requests.get(url)

        #if the request is successful         if response.status_code in range(200, 300):
            #extract the page html content for parsing the links             pageHtml = response.text
            pageUrl = response.url

            #call the parseLink function             parseLinks(pageHtml, pageUrl)

        else:
            print("Sorry Could not fetch the result status code {response.status_code}!")

    except:
        print(f"Could Not Connect to url {url}")