Using Google Jobs Listing Results API from SerpApi

Web Scraping (46 Part Series)

1 Scrape Google Shopping Tab with Python
2 Using Google Reverse Images API from SerpApi
… 42 more parts…
3 Scrape YouTube video page with Python
4 Scrape YouTube autocomplete results with Python
5 Scrape Google Realtime Search Trends with Python
6 Scrape Google Daily Search Trends with Python
7 Scrape Google Arts & Culture – Artists: All/A-Z/Time results with Python
8 Scrape Google Jobs organic results with Python
9 Scrape Google Play Games with Python
10 Scrape Google Play Apps with Python
11 Scrape Google Play Movies & TV with Python
12 Scrape Google Play Children (Kids) with Python
13 Scrape Google Play Books with Python
14 Scrape Brave Search Organic Results with Python
15 Scrape Brave News with Python
16 Scrape Brave Videos with Python
17 Scrape Brave Images with Python
18 Using Google Trends API from SerpApi
19 Using Google Jobs Listing Results API from SerpApi
20 Scrape Google Events Results with Python
21 Scrape Google Product Page with Python
22 Scrape Google Product Reviews Results with Python
23 Scrape Google Product Specs Results with Python
24 Scrape Google Product Online Sellers with Python
25 Using Google Product Local Sellers API from SerpApi
26 Using Google Maps Local Results API from SerpApi
27 Using Google Maps Place Results API from SerpApi
28 Using Google Maps Photos API from SerpApi
29 Using Google Maps Reviews API from SerpApi
30 Scrape Yelp Filters, Ad and Organic Results with Python
31 Using Yelp Reviews API from SerpApi with Python
32 Integrate The Home Depot Search Page Results Data with SerpApi and Python
33 How to Scrape Home Depot Product Data with SerpApi
34 Using Walmart Search Engine Results API from SerpApi
35 Using Walmart Product API from SerpApi
36 Scrape Google Lens with Python
37 SerpApi Changelog: December, 2022
38 Scraping Apple App Store Search with Python
39 Scraping Apple App Store Product Info And Reviews with Python
40 Scrape Google “Things to do” page with Python
41 Scrape Google Hotels with Python
42 Scraping Bing Organic Results using Python and SerpApi
43 How to Extract Bing News Data with SerpApi and Python
44 How to Extract Bing Images Data with SerpApi and Python
45 Scrape Google Flights with Python
46 Get Product Data from Bing Shopping with Python and SerpApi

What will be scraped

Note: Some queries may not display all sections. You can check your query in the playground.

Why using API?

No need to create a parser from scratch and maintain it.
Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
Pay for proxies, and CAPTCHA solvers.
Don’t need to use browser automation.

SerpApi handles everything on the backend with fast response times under ~2.5 seconds (~1.2 seconds with Ludicrous speed) per request and without browser automation, which becomes much faster. Response times and status rates are shown under SerpApi Status page.

Full Code

If you don’t need explanation, have a look at full code example in the online IDE.

from serpapi import GoogleSearch
import os, json


def extract_multiple_jobs():
    params = {
        # https://docs.python.org/3/library/os.html#os.getenv         'api_key': os.getenv('API_KEY'),        # your serpapi api         'engine': 'google_jobs',                # SerpApi search engine         'gl': 'us',                             # country of the search         'hl': 'en',                             # language of the search         'q': 'barista new york',                # search query     }

    search = GoogleSearch(params)               # where data extraction happens on the SerpApi backend     results = search.get_dict()                 # JSON -> Python dict 
    return [job.get('job_id') for job in results['jobs_results']]


def scrape_google_jobs_listing(job_ids):
    data = []

    for job_id in job_ids:
        params = {
            # https://docs.python.org/3/library/os.html#os.getenv             'api_key': os.getenv('API_KEY'),    # your serpapi api             'engine': 'google_jobs_listing',    # SerpApi search engine             'q': job_id,                        # search query (job_id)         }

        search = GoogleSearch(params)           # where data extraction happens on the SerpApi backend         results = search.get_dict()             # JSON -> Python dict 
        data.append({
            'job_id': job_id,
            'apply_options': results.get('apply_options'),
            'salaries': results.get('salaries'),
            'ratings': results.get('ratings')
        })

    return data


def main():
    job_ids = extract_multiple_jobs()
    google_jobs_listing_results = scrape_google_jobs_listing(job_ids)

    print(json.dumps(google_jobs_listing_results, indent=2, ensure_ascii=False))


if __name__ == '__main__':
    main()

Enter fullscreen mode Exit fullscreen mode

Preparation

Install library:

pip install google-search-results

Enter fullscreen mode Exit fullscreen mode

google-search-results is a SerpApi API package.

Code Explanation

Import libraries:

from serpapi import GoogleSearch
import os, json

Enter fullscreen mode Exit fullscreen mode

Library	Purpose
`GoogleSearch`	to scrape and parse Google results using SerpApi web scraping library.
`os`	to return environment variable (SerpApi API key) value.
`json`	to convert extracted data to a JSON object.

Top-level code environment

The extract_multiple_jobs() function is called to get all the job_id values. The resulting list of job_ids is passed to the scrape_google_jobs_listing(job_ids) function to retrieve the required data. The explanation of these functions will be in the corresponding headings below.

This code uses the generally accepted rule of using the __name__ == "__main__" construct:

def main():
    job_ids = extract_multiple_jobs()
    google_jobs_listing_results = scrape_google_jobs_listing(job_ids)

    print(json.dumps(google_jobs_listing_results, indent=2, ensure_ascii=False))


if __name__ == '__main__':
    main()

Enter fullscreen mode Exit fullscreen mode

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work.

You can watch the video Python Tutorial: if name == ‘main‘ for more details.

Extract Multiple Jobs

The function returns a list of job_id values. The value of this identifier will be used in the next function to create the request.

This function provides a code snippet for getting data from the first page. If you want to extract data using pagination, you can see it in the Scrape Google Jobs organic results with Python blog post.

At the beginning of the function, parameters are defined for generating the URL. If you want to pass other parameters to the URL, you can do so using the params dictionary.

params = {
    # https://docs.python.org/3/library/os.html#os.getenv     'api_key': os.getenv('API_KEY'),        # your serpapi api     'engine': 'google_jobs',                # SerpApi search engine     'gl': 'us',                             # country of the search     'hl': 'en',                             # language of the search     'q': 'barista new york',                # search query }

Enter fullscreen mode Exit fullscreen mode

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use.
`engine`	Set parameter to `google_jobs` to use the Google Jobs API engine.
`gl`	Parameter defines the country to use for the Google search. It’s a two-letter country code. (e.g., `us` for the United States, `uk` for United Kingdom, or `fr` for France). Head to the Google countries page for a full list of supported Google countries.
`hl`	Parameter defines the language to use for the Google Jobs search. It’s a two-letter language code. (e.g., `en` for English, `es` for Spanish, or `fr` for French). Head to the Google languages page for a full list of supported Google languages.
`q`	Parameter defines the query you want to search.

Then, we create a search object where the data is retrieved from the SerpApi backend. In the results dictionary we get data from JSON:

search = GoogleSearch(params)   # where data extraction happens on the SerpApi backend results = search.get_dict()     # JSON -> Python dict

Enter fullscreen mode Exit fullscreen mode

Returns a compiled list of all job_id using list comprehension:

return [job.get('job_id') for job in results['jobs_results']]

Enter fullscreen mode Exit fullscreen mode

The function looks like this:

def extract_multiple_jobs():
    params = {
        # https://docs.python.org/3/library/os.html#os.getenv         'api_key': os.getenv('API_KEY'),        # your serpapi api         'engine': 'google_jobs',                # SerpApi search engine         'gl': 'us',                             # country of the search         'hl': 'en',                             # language of the search         'q': 'barista new york',                # search query     }

    search = GoogleSearch(params)               # where data extraction happens on the SerpApi backend     results = search.get_dict()                 # JSON -> Python dict 
    return [job.get('job_id') for job in results['jobs_results']]

Enter fullscreen mode Exit fullscreen mode

Scrape Google Jobs Listing

This function takes the job_ids list and returns a list of all data.

Declaring the data list where the extracted data will be added:

data = []

Enter fullscreen mode Exit fullscreen mode

For each job_id value in the job_ids list, separate requests will be made and the corresponding data will be retrieved:

for job_id in job_ids:
    # data extraction will be here

Enter fullscreen mode Exit fullscreen mode

Next, we write a parameters for making a request:

params = {
    # https://docs.python.org/3/library/os.html#os.getenv     'api_key': os.getenv('API_KEY'),    # your serpapi api     'engine': 'google_jobs_listing',    # SerpApi search engine     'q': job_id,                        # search query (job_id) }

Enter fullscreen mode Exit fullscreen mode

Parameters	Explanation
`api_key`	Parameter defines the SerpApi private key to use.
`engine`	Set parameter to `google_jobs_listing` to use the Google Jobs Listing API engine.
`q`	Parameter defines the `job_id` string which can be obtained from Google Jobs API.

Then, we create a search object where the data is retrieved from the SerpApi backend. In the results dictionary we get data from JSON:

search = GoogleSearch(params)   # where data extraction happens on the SerpApi backend results = search.get_dict()     # JSON -> Python dict

Enter fullscreen mode Exit fullscreen mode

We can then create a dictionary structure from values such as job_id, apply_options, salaries, and ratings. The extracted data is written according to the corresponding keys. After that, the dictionary is appended to the data list:

data.append({
    'job_id': job_id,
    'apply_options': results.get('apply_options'),
    'salaries': results.get('salaries'),
    'ratings': results.get('ratings')
})

Enter fullscreen mode Exit fullscreen mode

At the end of the function, the data list is returned with the retrieved data for each job_id:

return data

Enter fullscreen mode Exit fullscreen mode

The complete function to scrape all data would look like this:

def scrape_google_jobs_listing(job_ids):
    data = []

    for job_id in job_ids:
        params = {
            # https://docs.python.org/3/library/os.html#os.getenv             'api_key': os.getenv('API_KEY'),    # your serpapi api             'engine': 'google_jobs_listing',    # SerpApi search engine             'q': job_id,                        # search query (job_id)         }

        search = GoogleSearch(params)           # where data extraction happens on the SerpApi backend         results = search.get_dict()             # JSON -> Python dict 
        data.append({
            'job_id': job_id,
            'apply_options': results.get('apply_options'),
            'salaries': results.get('salaries'),
            'ratings': results.get('ratings')
        })

    return data

Enter fullscreen mode Exit fullscreen mode

Output

[ { "job_id": "eyJqb2JfdGl0bGUiOiJCYXJpc3RhIiwiaHRpZG9jaWQiOiJuc3Y1d1hyNXdFOEFBQUFBQUFBQUFBPT0iLCJnbCI6InVzIiwiaGwiOiJlbiIsImZjIjoiRXVJQkNxSUJRVUYwVm14aVFtcFdYMjl0V0ZadU9USTNWV0ZZUlZRek9XRTJPVlJtYUc1RVZtaGpaRk5WT1VFMlNYZFpaR2ROU0dzdFoyMVBkMmxmUTNKS2RUQnJjMWxFT0dZNFNHWnFXRUZNTjB4eFRWVmtMV1JRVVRWaVJGbFVSMVo1YmxsVWVuazVPRzlxVVVsTmVXcFJjRXhPVWpWbWMwdFlTMlo2V21SUU1XSkZZa2hTY2pKaGRYcEdlRzVxTVVWNGIwZ3lhVXd3UlZGVVZ6Tk5XSGRNYXpKbVYyVjNFaGQzYkhCeFdTMWZUMHhNTW01d2RGRlFNRGhwUW05QmF4b2lRVVJWZVVWSFpqSTJWMjF3TjBoU2FtNDRPSHB5WkVWTldVMVhVWGRTU1hwMVFRIiwiZmN2IjoiMyIsImZjX2lkIjoiZmNfMSIsImFwcGx5X2xpbmsiOnsidGl0bGUiOiIubkZnMmVie2ZvbnQtd2VpZ2h0OjUwMH0uQmk2RGRje2ZvbnQtd2VpZ2h0OjUwMH1BcHBseSBkaXJlY3RseSBvbiBDdWxpbmFyeSBBZ2VudHMiLCJsaW5rIjoiaHR0cHM6Ly9jdWxpbmFyeWFnZW50cy5jb20vam9icy80MTc4NjMtQmFyaXN0YT91dG1fY2FtcGFpZ249Z29vZ2xlX2pvYnNfYXBwbHlcdTAwMjZ1dG1fc291cmNlPWdvb2dsZV9qb2JzX2FwcGx5XHUwMDI2dXRtX21lZGl1bT1vcmdhbmljIn19", "apply_options": [ { "title": "Apply on Trabajo.org", "link": "https://us.trabajo.org/job-1683-20221107-34e191c4eb8c8ca3ec69adfa55061df2?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic" }, { "title": "Apply on Jobs", "link": "https://us.fidanto.com/jobs/job-opening/nov-2022/barista-1432712052?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic" }, { "title": "Apply on Craigslist", "link": "https://newyork.craigslist.org/mnh/fbh/d/new-york-cafe-barista/7553733276.html?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic" }, { "title": "Apply directly on Culinary Agents", "link": "https://culinaryagents.com/jobs/417863-Barista?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic" } ], "salaries": null, "ratings": null }, ... other results ]

Enter fullscreen mode Exit fullscreen mode

Links

Join us on Twitter | YouTube

Add a Feature Request or a Bug

Web Scraping (46 Part Series)

原文链接：Using Google Jobs Listing Results API from SerpApi

展开阅读全文

文章版权声明 1、本网站名称：拾光赋
2、本站永久网址：https://www.blogs.ink
3、本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ：805375623进行删除处理。
4、本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5、本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6、本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。

THE END

Using Google Jobs Listing Results API from SerpApi

Web Scraping (46 Part Series)

What will be scraped

Why using API?

Full Code

Preparation

Code Explanation

Top-level code environment

Extract Multiple Jobs

Scrape Google Jobs Listing

Output

Links

Web Scraping (46 Part Series)

请登录后发表评论