Get Product Data from Bing Shopping with Python and SerpApi

Web Scraping (46 Part Series)

1 Scrape Google Shopping Tab with Python
2 Using Google Reverse Images API from SerpApi
… 42 more parts…
3 Scrape YouTube video page with Python
4 Scrape YouTube autocomplete results with Python
5 Scrape Google Realtime Search Trends with Python
6 Scrape Google Daily Search Trends with Python
7 Scrape Google Arts & Culture – Artists: All/A-Z/Time results with Python
8 Scrape Google Jobs organic results with Python
9 Scrape Google Play Games with Python
10 Scrape Google Play Apps with Python
11 Scrape Google Play Movies & TV with Python
12 Scrape Google Play Children (Kids) with Python
13 Scrape Google Play Books with Python
14 Scrape Brave Search Organic Results with Python
15 Scrape Brave News with Python
16 Scrape Brave Videos with Python
17 Scrape Brave Images with Python
18 Using Google Trends API from SerpApi
19 Using Google Jobs Listing Results API from SerpApi
20 Scrape Google Events Results with Python
21 Scrape Google Product Page with Python
22 Scrape Google Product Reviews Results with Python
23 Scrape Google Product Specs Results with Python
24 Scrape Google Product Online Sellers with Python
25 Using Google Product Local Sellers API from SerpApi
26 Using Google Maps Local Results API from SerpApi
27 Using Google Maps Place Results API from SerpApi
28 Using Google Maps Photos API from SerpApi
29 Using Google Maps Reviews API from SerpApi
30 Scrape Yelp Filters, Ad and Organic Results with Python
31 Using Yelp Reviews API from SerpApi with Python
32 Integrate The Home Depot Search Page Results Data with SerpApi and Python
33 How to Scrape Home Depot Product Data with SerpApi
34 Using Walmart Search Engine Results API from SerpApi
35 Using Walmart Product API from SerpApi
36 Scrape Google Lens with Python
37 SerpApi Changelog: December, 2022
38 Scraping Apple App Store Search with Python
39 Scraping Apple App Store Product Info And Reviews with Python
40 Scrape Google “Things to do” page with Python
41 Scrape Google Hotels with Python
42 Scraping Bing Organic Results using Python and SerpApi
43 How to Extract Bing News Data with SerpApi and Python
44 How to Extract Bing Images Data with SerpApi and Python
45 Scrape Google Flights with Python
46 Get Product Data from Bing Shopping with Python and SerpApi

• What will be scraped
• Full Code
• Preparation
• Code Explanation
  • Top-level code environment
  • Get inline shopping results
  • Get shopping results
• Output
• Links

What will be scraped

Note: Inline shopping results may not always be present.

Full Code

If you don’t need an explanation, have a look at the full code example in the online IDE.

<span>from</span> <span>selectolax.lexbor</span> <span>import</span> <span>LexborHTMLParser</span>
<span>import</span> <span>requests</span><span>,</span> <span>json</span><span>,</span> <span>re</span>
<span>def</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>:</span> <span>LexborHTMLParser</span><span>,</span> <span>product_counter</span><span>:</span> <span>int</span><span>)</span> <span>-></span> <span>list</span><span>:</span>
    <span>data</span> <span>=</span> <span>[]</span>
    <span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-gOffCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
        <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl span'</span><span>)</span>
        <span>raw_shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offDec'</span><span>)</span>
        <span>raw_rating</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.tags > span'</span><span>)</span>
        <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>product_link</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offLink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span>
        <span>brand</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>)</span> <span>else</span> <span>None</span>
        <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSlr'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span># https://regex101.com/r/lap8lr/1 </span>        <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
        <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>)</span> <span>else</span> <span>None</span>
        <span>shipping_text</span> <span>=</span> <span>raw_shipping</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_shipping</span> <span>else</span> <span>''</span>
        <span>shipping</span> <span>=</span> <span>shipping_text</span> <span>if</span> <span>'shipping'</span> <span>in</span> <span>shipping_text</span> <span>else</span> <span>None</span>
        <span>rating_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'aria-label'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
        <span># https://regex101.com/r/D2YTRI/1 </span>        <span>rating</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d.\d|\d]+'</span><span>,</span> <span>rating_text</span><span>).</span><span>group</span><span>(</span><span>0</span><span>))</span> <span>if</span> <span>rating_text</span> <span>else</span> <span>None</span>
        <span>reviews_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
        <span>reviews</span> <span>=</span> <span>reviews_text</span><span>.</span><span>replace</span><span>(</span><span>'('</span><span>,</span> <span>''</span><span>).</span><span>replace</span><span>(</span><span>')'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>reviews_text</span> <span>else</span> <span>None</span>
        <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>)</span> <span>else</span> <span>None</span>
        <span>data</span><span>.</span><span>append</span><span>({</span>
            <span>'position'</span><span>:</span> <span>position</span><span>,</span>
            <span>'title'</span><span>:</span> <span>title</span><span>,</span>
            <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
            <span>'brand'</span><span>:</span> <span>brand</span><span>,</span>
            <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
            <span>'price'</span><span>:</span> <span>price</span><span>,</span>
            <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
            <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
            <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
            <span>'rating'</span><span>:</span> <span>rating</span><span>,</span>
            <span>'reviews'</span><span>:</span> <span>reviews</span><span>,</span>
            <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
        <span>})</span>
    <span>return</span> <span>data</span>
<span>def</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>:</span> <span>LexborHTMLParser</span><span>,</span> <span>product_counter</span><span>:</span> <span>int</span><span>)</span> <span>-></span> <span>list</span><span>:</span>
    <span>data</span> <span>=</span> <span>[]</span>
    <span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-fullCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
        <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title span'</span><span>)</span>
        <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>product_link</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-titlelink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
        <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-seller'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.pd-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span># https://regex101.com/r/lap8lr/1 </span>        <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
        <span>sale</span> <span>=</span> <span>True</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-saletag'</span><span>)</span> <span>else</span> <span>False</span>
        <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>)</span> <span>else</span> <span>None</span>
        <span>shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>)</span> <span>else</span> <span>None</span>
        <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>)</span> <span>else</span> <span>None</span>
        <span>data</span><span>.</span><span>append</span><span>({</span>
            <span>'position'</span><span>:</span> <span>position</span><span>,</span>
            <span>'title'</span><span>:</span> <span>title</span><span>,</span>
            <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
            <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
            <span>'price'</span><span>:</span> <span>price</span><span>,</span>
            <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
            <span>'sale'</span><span>:</span> <span>sale</span><span>,</span>
            <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
            <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
            <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
        <span>})</span>
    <span>return</span> <span>data</span>
<span>def</span> <span>main</span><span>():</span>
    <span># https://docs.python-requests.org/en/master/user/quickstart/#custom-headers </span>    <span>headers</span> <span>=</span> <span>{</span>
        <span>'User-Agent'</span><span>:</span> <span>'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'</span>
    <span>}</span>
    <span>url</span> <span>=</span> <span>'https://www.bing.com/shop?q=ps4 controller'</span>
    <span>inline_shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
    <span>shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
    <span>bing_shopping_results</span> <span>=</span> <span>{</span>
        <span>'inline_shopping_results'</span><span>:</span> <span>[],</span>
        <span>'shopping_results'</span><span>:</span> <span>[]</span>
    <span>}</span>
    <span>while</span> <span>True</span><span>:</span>
        <span>html</span> <span>=</span> <span>requests</span><span>.</span><span>get</span><span>(</span><span>url</span><span>,</span> <span>headers</span><span>=</span><span>headers</span><span>)</span>
        <span>parser</span> <span>=</span> <span>LexborHTMLParser</span><span>(</span><span>html</span><span>.</span><span>text</span><span>)</span>
        <span>inline_shopping_page_results</span> <span>=</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>inline_shopping_results_counter</span><span>)</span>
        <span>bing_shopping_results</span><span>[</span><span>'inline_shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
        <span>shopping_page_results</span> <span>=</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>shopping_results_counter</span><span>)</span>
        <span>bing_shopping_results</span><span>[</span><span>'shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>shopping_page_results</span><span>)</span>
        <span>next_page_button</span> <span>=</span> <span>parser</span><span>.</span><span>css_first</span><span>(</span><span>'.sb_pagN_bp'</span><span>)</span>
        <span>if</span> <span>next_page_button</span><span>:</span>
            <span>url</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>next_page_button</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
            <span>inline_shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
            <span>shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>shopping_page_results</span><span>)</span>
        <span>else</span><span>:</span>
            <span>break</span>
    <span>print</span><span>(</span><span>json</span><span>.</span><span>dumps</span><span>(</span><span>bing_shopping_results</span><span>,</span> <span>indent</span><span>=</span><span>2</span><span>,</span> <span>ensure_ascii</span><span>=</span><span>False</span><span>))</span>
<span>if</span> <span>__name__</span> <span>==</span> <span>"__main__"</span><span>:</span>
    <span>main</span><span>()</span>
<span>from</span> <span>selectolax.lexbor</span> <span>import</span> <span>LexborHTMLParser</span>
<span>import</span> <span>requests</span><span>,</span> <span>json</span><span>,</span> <span>re</span>


<span>def</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>:</span> <span>LexborHTMLParser</span><span>,</span> <span>product_counter</span><span>:</span> <span>int</span><span>)</span> <span>-></span> <span>list</span><span>:</span>
    <span>data</span> <span>=</span> <span>[]</span>

    <span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-gOffCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
        <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl span'</span><span>)</span>
        <span>raw_shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offDec'</span><span>)</span>
        <span>raw_rating</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.tags > span'</span><span>)</span>

        <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>product_link</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offLink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span>
        <span>brand</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>)</span> <span>else</span> <span>None</span>
        <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSlr'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span># https://regex101.com/r/lap8lr/1 </span>        <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
        <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>)</span> <span>else</span> <span>None</span>
        <span>shipping_text</span> <span>=</span> <span>raw_shipping</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_shipping</span> <span>else</span> <span>''</span>
        <span>shipping</span> <span>=</span> <span>shipping_text</span> <span>if</span> <span>'shipping'</span> <span>in</span> <span>shipping_text</span> <span>else</span> <span>None</span>
        <span>rating_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'aria-label'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
        <span># https://regex101.com/r/D2YTRI/1 </span>        <span>rating</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d.\d|\d]+'</span><span>,</span> <span>rating_text</span><span>).</span><span>group</span><span>(</span><span>0</span><span>))</span> <span>if</span> <span>rating_text</span> <span>else</span> <span>None</span>
        <span>reviews_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
        <span>reviews</span> <span>=</span> <span>reviews_text</span><span>.</span><span>replace</span><span>(</span><span>'('</span><span>,</span> <span>''</span><span>).</span><span>replace</span><span>(</span><span>')'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>reviews_text</span> <span>else</span> <span>None</span>
        <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>)</span> <span>else</span> <span>None</span>

        <span>data</span><span>.</span><span>append</span><span>({</span>
            <span>'position'</span><span>:</span> <span>position</span><span>,</span>
            <span>'title'</span><span>:</span> <span>title</span><span>,</span>
            <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
            <span>'brand'</span><span>:</span> <span>brand</span><span>,</span>
            <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
            <span>'price'</span><span>:</span> <span>price</span><span>,</span>
            <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
            <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
            <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
            <span>'rating'</span><span>:</span> <span>rating</span><span>,</span>
            <span>'reviews'</span><span>:</span> <span>reviews</span><span>,</span>
            <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
        <span>})</span>

    <span>return</span> <span>data</span>


<span>def</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>:</span> <span>LexborHTMLParser</span><span>,</span> <span>product_counter</span><span>:</span> <span>int</span><span>)</span> <span>-></span> <span>list</span><span>:</span>
    <span>data</span> <span>=</span> <span>[]</span>

    <span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-fullCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
        <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title span'</span><span>)</span>

        <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>product_link</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-titlelink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
        <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-seller'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.pd-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span># https://regex101.com/r/lap8lr/1 </span>        <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
        <span>sale</span> <span>=</span> <span>True</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-saletag'</span><span>)</span> <span>else</span> <span>False</span>
        <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>)</span> <span>else</span> <span>None</span>
        <span>shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>)</span> <span>else</span> <span>None</span>
        <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>)</span> <span>else</span> <span>None</span>

        <span>data</span><span>.</span><span>append</span><span>({</span>
            <span>'position'</span><span>:</span> <span>position</span><span>,</span>
            <span>'title'</span><span>:</span> <span>title</span><span>,</span>
            <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
            <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
            <span>'price'</span><span>:</span> <span>price</span><span>,</span>
            <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
            <span>'sale'</span><span>:</span> <span>sale</span><span>,</span>
            <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
            <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
            <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
        <span>})</span>

    <span>return</span> <span>data</span>


<span>def</span> <span>main</span><span>():</span>
    <span># https://docs.python-requests.org/en/master/user/quickstart/#custom-headers </span>    <span>headers</span> <span>=</span> <span>{</span>
        <span>'User-Agent'</span><span>:</span> <span>'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'</span>
    <span>}</span>

    <span>url</span> <span>=</span> <span>'https://www.bing.com/shop?q=ps4 controller'</span>

    <span>inline_shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
    <span>shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>

    <span>bing_shopping_results</span> <span>=</span> <span>{</span>
        <span>'inline_shopping_results'</span><span>:</span> <span>[],</span>
        <span>'shopping_results'</span><span>:</span> <span>[]</span>
    <span>}</span>

    <span>while</span> <span>True</span><span>:</span>
        <span>html</span> <span>=</span> <span>requests</span><span>.</span><span>get</span><span>(</span><span>url</span><span>,</span> <span>headers</span><span>=</span><span>headers</span><span>)</span>
        <span>parser</span> <span>=</span> <span>LexborHTMLParser</span><span>(</span><span>html</span><span>.</span><span>text</span><span>)</span>

        <span>inline_shopping_page_results</span> <span>=</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>inline_shopping_results_counter</span><span>)</span>
        <span>bing_shopping_results</span><span>[</span><span>'inline_shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>

        <span>shopping_page_results</span> <span>=</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>shopping_results_counter</span><span>)</span>
        <span>bing_shopping_results</span><span>[</span><span>'shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>shopping_page_results</span><span>)</span>

        <span>next_page_button</span> <span>=</span> <span>parser</span><span>.</span><span>css_first</span><span>(</span><span>'.sb_pagN_bp'</span><span>)</span>

        <span>if</span> <span>next_page_button</span><span>:</span>
            <span>url</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>next_page_button</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
            <span>inline_shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
            <span>shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>shopping_page_results</span><span>)</span>
        <span>else</span><span>:</span>
            <span>break</span>

    <span>print</span><span>(</span><span>json</span><span>.</span><span>dumps</span><span>(</span><span>bing_shopping_results</span><span>,</span> <span>indent</span><span>=</span><span>2</span><span>,</span> <span>ensure_ascii</span><span>=</span><span>False</span><span>))</span>


<span>if</span> <span>__name__</span> <span>==</span> <span>"__main__"</span><span>:</span>
    <span>main</span><span>()</span>
from selectolax.lexbor import LexborHTMLParser
import requests, json, re


def get_inline_shopping_results(parser: LexborHTMLParser, product_counter: int) -> list:
    data = []

    for position, product in enumerate(parser.root.css('.br-gOffCard'), start=product_counter):
        raw_title = product.css_first('.br-offTtl span')
        raw_shipping = product.css_first('.br-offDec')
        raw_rating = product.css_first('.tags > span')

        title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-offTtl').text().strip()
        product_link = product.css_first('.br-offLink').attributes.get('href', '')
        brand = product.css_first('.br-offSecLbl').text().strip() if product.css_first('.br-offSecLbl') else None
        seller = product.css_first('.br-offSlr').text().strip()
        price = product.css_first('.br-price').text().strip()
        # https://regex101.com/r/lap8lr/1         extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
        old_price = product.css_first('.br-standardPriceDemoted').text().strip() if product.css_first('.br-standardPriceDemoted') else None
        shipping_text = raw_shipping.text().strip() if raw_shipping else ''
        shipping = shipping_text if 'shipping' in shipping_text else None
        rating_text = raw_rating.attributes.get('aria-label', '') if raw_rating else None
        # https://regex101.com/r/D2YTRI/1         rating = float(re.search(r'[\d.\d|\d]+', rating_text).group(0)) if rating_text else None
        reviews_text = raw_rating.text().strip() if raw_rating else None
        reviews = reviews_text.replace('(', '').replace(')', '') if reviews_text else None
        thumbnail = product.css_first('.cico img').attributes.get('src', None) if product.css_first('.cico img') else None

        data.append({
            'position': position,
            'title': title,
            'product_link': product_link,
            'brand': brand,
            'seller': seller,
            'price': price,
            'extracted_price': extracted_price,
            'old_price': old_price,
            'shipping': shipping,
            'rating': rating,
            'reviews': reviews,
            'thumbnail': thumbnail
        })

    return data


def get_shopping_results(parser: LexborHTMLParser, product_counter: int) -> list:
    data = []

    for position, product in enumerate(parser.root.css('.br-fullCard'), start=product_counter):
        raw_title = product.css_first('.br-title span')

        title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-title').text().strip()
        product_link = f"https://www.bing.com{product.css_first('.br-titlelink').attributes.get('href', '')}"
        seller = product.css_first('.br-seller').text().strip()
        price = product.css_first('.pd-price').text().strip()
        # https://regex101.com/r/lap8lr/1         extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
        sale = True if product.css_first('.br-saletag') else False
        old_price = product.css_first('.br-pdOfferPrice').text().strip() if product.css_first('.br-pdOfferPrice') else None
        shipping = product.css_first('.br-decoSlot').text().strip() if product.css_first('.br-decoSlot') else None
        thumbnail = product.css_first('.br-pdMainImg img').attributes.get('src', None) if product.css_first('.br-pdMainImg img') else None

        data.append({
            'position': position,
            'title': title,
            'product_link': product_link,
            'seller': seller,
            'price': price,
            'extracted_price': extracted_price,
            'sale': sale,
            'old_price': old_price,
            'shipping': shipping,
            'thumbnail': thumbnail
        })

    return data


def main():
    # https://docs.python-requests.org/en/master/user/quickstart/#custom-headers     headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'
    }

    url = 'https://www.bing.com/shop?q=ps4 controller'

    inline_shopping_results_counter: int = 1
    shopping_results_counter: int = 1

    bing_shopping_results = {
        'inline_shopping_results': [],
        'shopping_results': []
    }

    while True:
        html = requests.get(url, headers=headers)
        parser = LexborHTMLParser(html.text)

        inline_shopping_page_results = get_inline_shopping_results(parser, inline_shopping_results_counter)
        bing_shopping_results['inline_shopping_results'].extend(inline_shopping_page_results)

        shopping_page_results = get_shopping_results(parser, shopping_results_counter)
        bing_shopping_results['shopping_results'].extend(shopping_page_results)

        next_page_button = parser.css_first('.sb_pagN_bp')

        if next_page_button:
            url = f"https://www.bing.com{next_page_button.attributes.get('href', '')}"
            inline_shopping_results_counter += len(inline_shopping_page_results)
            shopping_results_counter += len(shopping_page_results)
        else:
            break

    print(json.dumps(bing_shopping_results, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

Enter fullscreen mode Exit fullscreen mode

Preparation

Install libraries

pip install requests selectolax
pip install requests selectolax
pip install requests selectolax

Enter fullscreen mode Exit fullscreen mode

Reduce the chance of being blocked

Make sure you’re using request headers user-agent to act as a “real” user visit. Because default requests user-agent is python-requests and websites understand that it’s most likely a script that sends a request. Check what’s your user-agent.

There’s a how to reduce the chance of being blocked while web scraping blog post that can get you familiar with basic and more advanced approaches.

Code Explanation

Import libraries:

<span>from</span> <span>selectolax.lexbor</span> <span>import</span> <span>LexborHTMLParser</span>
<span>import</span> <span>requests</span><span>,</span> <span>json</span><span>,</span> <span>re</span>
<span>from</span> <span>selectolax.lexbor</span> <span>import</span> <span>LexborHTMLParser</span>
<span>import</span> <span>requests</span><span>,</span> <span>json</span><span>,</span> <span>re</span>
from selectolax.lexbor import LexborHTMLParser
import requests, json, re

Enter fullscreen mode Exit fullscreen mode

Library	Purpose
LexborHTMLParser	a fast HTML5 parser with CSS selectors using Lexbor engine.
requests	to make a request to the website.
json	to convert extracted data to a JSON object.
re	to extract parts of the data via regular expression.

The next part of the code is divided into functions. Each function is described in the corresponding heading below.

Top-level code environment

The request headers are generated:

<span># https://docs.python-requests.org/en/master/user/quickstart/#custom-headers </span><span>headers</span> <span>=</span> <span>{</span>
    <span>'User-Agent'</span><span>:</span> <span>'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'</span>
<span>}</span>
<span># https://docs.python-requests.org/en/master/user/quickstart/#custom-headers </span><span>headers</span> <span>=</span> <span>{</span>
    <span>'User-Agent'</span><span>:</span> <span>'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'</span>
<span>}</span>
# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'
}

Enter fullscreen mode Exit fullscreen mode

The url variable contains a link to the Bing Shopping page:

<span>url</span> <span>=</span> <span>'https://www.bing.com/shop?q=ps4 controller'</span>
<span>url</span> <span>=</span> <span>'https://www.bing.com/shop?q=ps4 controller'</span>
url = 'https://www.bing.com/shop?q=ps4 controller'

Enter fullscreen mode Exit fullscreen mode

Variables are created to determine the position of the current product, taking into account pagination. For example, if on the first page the position of the last product is 36, then on the second page the position of the first product will be 37 (not 1):

<span>inline_shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
<span>shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
<span>inline_shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
<span>shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
inline_shopping_results_counter: int = 1
shopping_results_counter: int = 1

Enter fullscreen mode Exit fullscreen mode

The bing_shopping_results dictionary is formed, where data on the corresponding keys will subsequently be added:

<span>bing_shopping_results</span> <span>=</span> <span>{</span>
    <span>'inline_shopping_results'</span><span>:</span> <span>[],</span>
    <span>'shopping_results'</span><span>:</span> <span>[]</span>
<span>}</span>
<span>bing_shopping_results</span> <span>=</span> <span>{</span>
    <span>'inline_shopping_results'</span><span>:</span> <span>[],</span>
    <span>'shopping_results'</span><span>:</span> <span>[]</span>
<span>}</span>
bing_shopping_results = {
    'inline_shopping_results': [],
    'shopping_results': []
}

Enter fullscreen mode Exit fullscreen mode

Requests are sent in a loop for each page:

<span>while</span> <span>True</span><span>:</span>
    <span># pagination </span>
<span>while</span> <span>True</span><span>:</span>
    <span># pagination </span>
while True:
    # pagination 

Enter fullscreen mode Exit fullscreen mode

Make a request, pass url and headers. The data extraction itself is done with selectolax because it has Lexbor parser which is incredibly fast, like 186% faster compared to bs4 with lxml backend when parsing data with 3000 iterations 5 times:

<span>html</span> <span>=</span> <span>requests</span><span>.</span><span>get</span><span>(</span><span>url</span><span>,</span> <span>headers</span><span>=</span><span>headers</span><span>)</span>
<span>parser</span> <span>=</span> <span>LexborHTMLParser</span><span>(</span><span>html</span><span>.</span><span>text</span><span>)</span>
<span>html</span> <span>=</span> <span>requests</span><span>.</span><span>get</span><span>(</span><span>url</span><span>,</span> <span>headers</span><span>=</span><span>headers</span><span>)</span>
<span>parser</span> <span>=</span> <span>LexborHTMLParser</span><span>(</span><span>html</span><span>.</span><span>text</span><span>)</span>
html = requests.get(url, headers=headers)
parser = LexborHTMLParser(html.text)

Enter fullscreen mode Exit fullscreen mode

By calling the appropriate functions, all the necessary data is retrieved and written to the bing_shopping_results dictionary. These functions are detailed in the sections below.

<span>inline_shopping_page_results</span> <span>=</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>inline_shopping_results_counter</span><span>)</span>
<span>bing_shopping_results</span><span>[</span><span>'inline_shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
<span>shopping_page_results</span> <span>=</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>shopping_results_counter</span><span>)</span>
<span>bing_shopping_results</span><span>[</span><span>'shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>shopping_page_results</span><span>)</span>
<span>inline_shopping_page_results</span> <span>=</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>inline_shopping_results_counter</span><span>)</span>
<span>bing_shopping_results</span><span>[</span><span>'inline_shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>

<span>shopping_page_results</span> <span>=</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>shopping_results_counter</span><span>)</span>
<span>bing_shopping_results</span><span>[</span><span>'shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>shopping_page_results</span><span>)</span>
inline_shopping_page_results = get_inline_shopping_results(parser, inline_shopping_results_counter)
bing_shopping_results['inline_shopping_results'].extend(inline_shopping_page_results)

shopping_page_results = get_shopping_results(parser, shopping_results_counter)
bing_shopping_results['shopping_results'].extend(shopping_page_results)

Enter fullscreen mode Exit fullscreen mode

At the end of the iteration, it checks whether the “Next” button is present. If such a button is present, then the url and counters are updated. Otherwise, the loop is stopped.

<span>next_page_button</span> <span>=</span> <span>parser</span><span>.</span><span>css_first</span><span>(</span><span>'.sb_pagN_bp'</span><span>)</span>
<span>if</span> <span>next_page_button</span><span>:</span>
    <span>url</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>next_page_button</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
    <span>inline_shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
    <span>shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>shopping_page_results</span><span>)</span>
<span>else</span><span>:</span>
    <span>break</span>
<span>next_page_button</span> <span>=</span> <span>parser</span><span>.</span><span>css_first</span><span>(</span><span>'.sb_pagN_bp'</span><span>)</span>

<span>if</span> <span>next_page_button</span><span>:</span>
    <span>url</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>next_page_button</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
    <span>inline_shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
    <span>shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>shopping_page_results</span><span>)</span>
<span>else</span><span>:</span>
    <span>break</span>
next_page_button = parser.css_first('.sb_pagN_bp')

if next_page_button:
    url = f"https://www.bing.com{next_page_button.attributes.get('href', '')}"
    inline_shopping_results_counter += len(inline_shopping_page_results)
    shopping_results_counter += len(shopping_page_results)
else:
    break

Enter fullscreen mode Exit fullscreen mode

After receiving the data from all pages, they are output in JSON format:

<span>print</span><span>(</span><span>json</span><span>.</span><span>dumps</span><span>(</span><span>bing_shopping_results</span><span>,</span> <span>indent</span><span>=</span><span>2</span><span>,</span> <span>ensure_ascii</span><span>=</span><span>False</span><span>))</span>
<span>print</span><span>(</span><span>json</span><span>.</span><span>dumps</span><span>(</span><span>bing_shopping_results</span><span>,</span> <span>indent</span><span>=</span><span>2</span><span>,</span> <span>ensure_ascii</span><span>=</span><span>False</span><span>))</span>
print(json.dumps(bing_shopping_results, indent=2, ensure_ascii=False))

Enter fullscreen mode Exit fullscreen mode

This code uses boilerplate __name__ == "__main__" construct that protects users from accidentally invoking the script when they didn’t intend to. This indicates that the code is a runnable script:

<span>def</span> <span>main</span><span>():</span>
    <span># https://docs.python-requests.org/en/master/user/quickstart/#custom-headers </span>    <span>headers</span> <span>=</span> <span>{</span>
        <span>'User-Agent'</span><span>:</span> <span>'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'</span>
    <span>}</span>
    <span>url</span> <span>=</span> <span>'https://www.bing.com/shop?q=ps4 controller'</span>
    <span>inline_shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
    <span>shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
    <span>bing_shopping_results</span> <span>=</span> <span>{</span>
        <span>'inline_shopping_results'</span><span>:</span> <span>[],</span>
        <span>'shopping_results'</span><span>:</span> <span>[]</span>
    <span>}</span>
    <span>while</span> <span>True</span><span>:</span>
        <span>html</span> <span>=</span> <span>requests</span><span>.</span><span>get</span><span>(</span><span>url</span><span>,</span> <span>headers</span><span>=</span><span>headers</span><span>)</span>
        <span>parser</span> <span>=</span> <span>LexborHTMLParser</span><span>(</span><span>html</span><span>.</span><span>text</span><span>)</span>
        <span>inline_shopping_page_results</span> <span>=</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>inline_shopping_results_counter</span><span>)</span>
        <span>bing_shopping_results</span><span>[</span><span>'inline_shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
        <span>shopping_page_results</span> <span>=</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>shopping_results_counter</span><span>)</span>
        <span>bing_shopping_results</span><span>[</span><span>'shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>shopping_page_results</span><span>)</span>
        <span>next_page_button</span> <span>=</span> <span>parser</span><span>.</span><span>css_first</span><span>(</span><span>'.sb_pagN_bp'</span><span>)</span>
        <span>if</span> <span>next_page_button</span><span>:</span>
            <span>url</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>next_page_button</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
            <span>inline_shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
            <span>shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>shopping_page_results</span><span>)</span>
        <span>else</span><span>:</span>
            <span>break</span>
    <span>print</span><span>(</span><span>json</span><span>.</span><span>dumps</span><span>(</span><span>bing_shopping_results</span><span>,</span> <span>indent</span><span>=</span><span>2</span><span>,</span> <span>ensure_ascii</span><span>=</span><span>False</span><span>))</span>
<span>if</span> <span>__name__</span> <span>==</span> <span>"__main__"</span><span>:</span>
    <span>main</span><span>()</span>
<span>def</span> <span>main</span><span>():</span>
    <span># https://docs.python-requests.org/en/master/user/quickstart/#custom-headers </span>    <span>headers</span> <span>=</span> <span>{</span>
        <span>'User-Agent'</span><span>:</span> <span>'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'</span>
    <span>}</span>

    <span>url</span> <span>=</span> <span>'https://www.bing.com/shop?q=ps4 controller'</span>

    <span>inline_shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>
    <span>shopping_results_counter</span><span>:</span> <span>int</span> <span>=</span> <span>1</span>

    <span>bing_shopping_results</span> <span>=</span> <span>{</span>
        <span>'inline_shopping_results'</span><span>:</span> <span>[],</span>
        <span>'shopping_results'</span><span>:</span> <span>[]</span>
    <span>}</span>

    <span>while</span> <span>True</span><span>:</span>
        <span>html</span> <span>=</span> <span>requests</span><span>.</span><span>get</span><span>(</span><span>url</span><span>,</span> <span>headers</span><span>=</span><span>headers</span><span>)</span>
        <span>parser</span> <span>=</span> <span>LexborHTMLParser</span><span>(</span><span>html</span><span>.</span><span>text</span><span>)</span>

        <span>inline_shopping_page_results</span> <span>=</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>inline_shopping_results_counter</span><span>)</span>
        <span>bing_shopping_results</span><span>[</span><span>'inline_shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>

        <span>shopping_page_results</span> <span>=</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>,</span> <span>shopping_results_counter</span><span>)</span>
        <span>bing_shopping_results</span><span>[</span><span>'shopping_results'</span><span>].</span><span>extend</span><span>(</span><span>shopping_page_results</span><span>)</span>

        <span>next_page_button</span> <span>=</span> <span>parser</span><span>.</span><span>css_first</span><span>(</span><span>'.sb_pagN_bp'</span><span>)</span>

        <span>if</span> <span>next_page_button</span><span>:</span>
            <span>url</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>next_page_button</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
            <span>inline_shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>inline_shopping_page_results</span><span>)</span>
            <span>shopping_results_counter</span> <span>+=</span> <span>len</span><span>(</span><span>shopping_page_results</span><span>)</span>
        <span>else</span><span>:</span>
            <span>break</span>

    <span>print</span><span>(</span><span>json</span><span>.</span><span>dumps</span><span>(</span><span>bing_shopping_results</span><span>,</span> <span>indent</span><span>=</span><span>2</span><span>,</span> <span>ensure_ascii</span><span>=</span><span>False</span><span>))</span>


<span>if</span> <span>__name__</span> <span>==</span> <span>"__main__"</span><span>:</span>
    <span>main</span><span>()</span>
def main():
    # https://docs.python-requests.org/en/master/user/quickstart/#custom-headers     headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36'
    }

    url = 'https://www.bing.com/shop?q=ps4 controller'

    inline_shopping_results_counter: int = 1
    shopping_results_counter: int = 1

    bing_shopping_results = {
        'inline_shopping_results': [],
        'shopping_results': []
    }

    while True:
        html = requests.get(url, headers=headers)
        parser = LexborHTMLParser(html.text)

        inline_shopping_page_results = get_inline_shopping_results(parser, inline_shopping_results_counter)
        bing_shopping_results['inline_shopping_results'].extend(inline_shopping_page_results)

        shopping_page_results = get_shopping_results(parser, shopping_results_counter)
        bing_shopping_results['shopping_results'].extend(shopping_page_results)

        next_page_button = parser.css_first('.sb_pagN_bp')

        if next_page_button:
            url = f"https://www.bing.com{next_page_button.attributes.get('href', '')}"
            inline_shopping_results_counter += len(inline_shopping_page_results)
            shopping_results_counter += len(shopping_page_results)
        else:
            break

    print(json.dumps(bing_shopping_results, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

Enter fullscreen mode Exit fullscreen mode

This check will only be performed if the user has run this file. If the user imports this file into another, then the check will not work.

You can watch the video Python Tutorial: if name == ‘main‘ for more details.

Get inline shopping results

The function takes a parser and a product_counter. Returns a list with the retrieved data.

The data list is declared to which the extracted data will be added:

<span>data</span> <span>=</span> <span>[]</span>
<span>data</span> <span>=</span> <span>[]</span>
data = []

Enter fullscreen mode Exit fullscreen mode

To retrieve all products from a page, you need to use the css() method and pass the .br-gOffCard selector there.

After that, you need to iterate over the resulting list of products using the for loop with the built-in enumerate() function. This function adds a counter to the iterable and returns it. The counter is needed to assign a position to each product:

<span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-gOffCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
    <span># data extraction </span>
<span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-gOffCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
    <span># data extraction </span>
for position, product in enumerate(parser.root.css('.br-gOffCard'), start=product_counter):
    # data extraction 

Enter fullscreen mode Exit fullscreen mode

Retrieving the required data is done as follows: we find the required selector for current product using the css_first() method, after this we extract the text value or attribute value:

<span>product_link</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offLink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span>
<span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSlr'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
<span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
<span>product_link</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offLink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span>
<span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSlr'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
<span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
product_link = product.css_first('.br-offLink').attributes.get('href', '')
seller = product.css_first('.br-offSlr').text().strip()
price = product.css_first('.br-price').text().strip()

Enter fullscreen mode Exit fullscreen mode

There are data that may not be present in some products. In this case, checks are added:

<span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl span'</span><span>)</span>
<span>raw_shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offDec'</span><span>)</span>
<span>raw_rating</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.tags > span'</span><span>)</span>
<span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
<span>brand</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>)</span> <span>else</span> <span>None</span>
<span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>)</span> <span>else</span> <span>None</span>
<span>shipping_text</span> <span>=</span> <span>raw_shipping</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_shipping</span> <span>else</span> <span>''</span>
<span>shipping</span> <span>=</span> <span>shipping_text</span> <span>if</span> <span>'shipping'</span> <span>in</span> <span>shipping_text</span> <span>else</span> <span>None</span>
<span>rating_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'aria-label'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
<span>reviews_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
<span>reviews</span> <span>=</span> <span>reviews_text</span><span>.</span><span>replace</span><span>(</span><span>'('</span><span>,</span> <span>''</span><span>).</span><span>replace</span><span>(</span><span>')'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>reviews_text</span> <span>else</span> <span>None</span>
<span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>)</span> <span>else</span> <span>None</span>
<span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl span'</span><span>)</span>
<span>raw_shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offDec'</span><span>)</span>
<span>raw_rating</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.tags > span'</span><span>)</span>

<span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
<span>brand</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>)</span> <span>else</span> <span>None</span>
<span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>)</span> <span>else</span> <span>None</span>
<span>shipping_text</span> <span>=</span> <span>raw_shipping</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_shipping</span> <span>else</span> <span>''</span>
<span>shipping</span> <span>=</span> <span>shipping_text</span> <span>if</span> <span>'shipping'</span> <span>in</span> <span>shipping_text</span> <span>else</span> <span>None</span>
<span>rating_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'aria-label'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
<span>reviews_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
<span>reviews</span> <span>=</span> <span>reviews_text</span><span>.</span><span>replace</span><span>(</span><span>'('</span><span>,</span> <span>''</span><span>).</span><span>replace</span><span>(</span><span>')'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>reviews_text</span> <span>else</span> <span>None</span>
<span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>)</span> <span>else</span> <span>None</span>
raw_title = product.css_first('.br-offTtl span')
raw_shipping = product.css_first('.br-offDec')
raw_rating = product.css_first('.tags > span')

title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-offTtl').text().strip()
brand = product.css_first('.br-offSecLbl').text().strip() if product.css_first('.br-offSecLbl') else None
old_price = product.css_first('.br-standardPriceDemoted').text().strip() if product.css_first('.br-standardPriceDemoted') else None
shipping_text = raw_shipping.text().strip() if raw_shipping else ''
shipping = shipping_text if 'shipping' in shipping_text else None
rating_text = raw_rating.attributes.get('aria-label', '') if raw_rating else None
reviews_text = raw_rating.text().strip() if raw_rating else None
reviews = reviews_text.replace('(', '').replace(')', '') if reviews_text else None
thumbnail = product.css_first('.cico img').attributes.get('src', None) if product.css_first('.cico img') else None

Enter fullscreen mode Exit fullscreen mode

If you need to get a specific value, then you can use regular expression to extract them:

<span># https://regex101.com/r/lap8lr/1 </span><span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
<span># https://regex101.com/r/D2YTRI/1 </span><span>rating</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d.\d|\d]+'</span><span>,</span> <span>rating_text</span><span>).</span><span>group</span><span>(</span><span>0</span><span>))</span> <span>if</span> <span>rating_text</span> <span>else</span> <span>None</span>
<span># https://regex101.com/r/lap8lr/1 </span><span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
<span># https://regex101.com/r/D2YTRI/1 </span><span>rating</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d.\d|\d]+'</span><span>,</span> <span>rating_text</span><span>).</span><span>group</span><span>(</span><span>0</span><span>))</span> <span>if</span> <span>rating_text</span> <span>else</span> <span>None</span>
# https://regex101.com/r/lap8lr/1 extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
# https://regex101.com/r/D2YTRI/1 rating = float(re.search(r'[\d.\d|\d]+', rating_text).group(0)) if rating_text else None

Enter fullscreen mode Exit fullscreen mode

At the end of each iteration, product data is added to the data list:

<span>data</span><span>.</span><span>append</span><span>({</span>
    <span>'position'</span><span>:</span> <span>position</span><span>,</span>
    <span>'title'</span><span>:</span> <span>title</span><span>,</span>
    <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
    <span>'brand'</span><span>:</span> <span>brand</span><span>,</span>
    <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
    <span>'price'</span><span>:</span> <span>price</span><span>,</span>
    <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
    <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
    <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
    <span>'rating'</span><span>:</span> <span>rating</span><span>,</span>
    <span>'reviews'</span><span>:</span> <span>reviews</span><span>,</span>
    <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
<span>})</span>
<span>data</span><span>.</span><span>append</span><span>({</span>
    <span>'position'</span><span>:</span> <span>position</span><span>,</span>
    <span>'title'</span><span>:</span> <span>title</span><span>,</span>
    <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
    <span>'brand'</span><span>:</span> <span>brand</span><span>,</span>
    <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
    <span>'price'</span><span>:</span> <span>price</span><span>,</span>
    <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
    <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
    <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
    <span>'rating'</span><span>:</span> <span>rating</span><span>,</span>
    <span>'reviews'</span><span>:</span> <span>reviews</span><span>,</span>
    <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
<span>})</span>
data.append({
    'position': position,
    'title': title,
    'product_link': product_link,
    'brand': brand,
    'seller': seller,
    'price': price,
    'extracted_price': extracted_price,
    'old_price': old_price,
    'shipping': shipping,
    'rating': rating,
    'reviews': reviews,
    'thumbnail': thumbnail
})

Enter fullscreen mode Exit fullscreen mode

After all the operations are done, return the data list:

<span>return</span> <span>data</span>
<span>return</span> <span>data</span>
return data

Enter fullscreen mode Exit fullscreen mode

The function looks like this:

<span>def</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>:</span> <span>LexborHTMLParser</span><span>,</span> <span>product_counter</span><span>:</span> <span>int</span><span>)</span> <span>-></span> <span>list</span><span>:</span>
    <span>data</span> <span>=</span> <span>[]</span>
    <span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-gOffCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
        <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl span'</span><span>)</span>
        <span>raw_shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offDec'</span><span>)</span>
        <span>raw_rating</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.tags > span'</span><span>)</span>
        <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>product_link</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offLink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span>
        <span>brand</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>)</span> <span>else</span> <span>None</span>
        <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSlr'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span># https://regex101.com/r/lap8lr/1 </span>        <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
        <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>)</span> <span>else</span> <span>None</span>
        <span>shipping_text</span> <span>=</span> <span>raw_shipping</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_shipping</span> <span>else</span> <span>''</span>
        <span>shipping</span> <span>=</span> <span>shipping_text</span> <span>if</span> <span>'shipping'</span> <span>in</span> <span>shipping_text</span> <span>else</span> <span>None</span>
        <span>rating_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'aria-label'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
        <span># https://regex101.com/r/D2YTRI/1 </span>        <span>rating</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d.\d|\d]+'</span><span>,</span> <span>rating_text</span><span>).</span><span>group</span><span>(</span><span>0</span><span>))</span> <span>if</span> <span>rating_text</span> <span>else</span> <span>None</span>
        <span>reviews_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
        <span>reviews</span> <span>=</span> <span>reviews_text</span><span>.</span><span>replace</span><span>(</span><span>'('</span><span>,</span> <span>''</span><span>).</span><span>replace</span><span>(</span><span>')'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>reviews_text</span> <span>else</span> <span>None</span>
        <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>)</span> <span>else</span> <span>None</span>
        <span>data</span><span>.</span><span>append</span><span>({</span>
            <span>'position'</span><span>:</span> <span>position</span><span>,</span>
            <span>'title'</span><span>:</span> <span>title</span><span>,</span>
            <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
            <span>'brand'</span><span>:</span> <span>brand</span><span>,</span>
            <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
            <span>'price'</span><span>:</span> <span>price</span><span>,</span>
            <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
            <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
            <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
            <span>'rating'</span><span>:</span> <span>rating</span><span>,</span>
            <span>'reviews'</span><span>:</span> <span>reviews</span><span>,</span>
            <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
        <span>})</span>
    <span>return</span> <span>data</span>
<span>def</span> <span>get_inline_shopping_results</span><span>(</span><span>parser</span><span>:</span> <span>LexborHTMLParser</span><span>,</span> <span>product_counter</span><span>:</span> <span>int</span><span>)</span> <span>-></span> <span>list</span><span>:</span>
    <span>data</span> <span>=</span> <span>[]</span>

    <span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-gOffCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
        <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl span'</span><span>)</span>
        <span>raw_shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offDec'</span><span>)</span>
        <span>raw_rating</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.tags > span'</span><span>)</span>

        <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offTtl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>product_link</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offLink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span>
        <span>brand</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSecLbl'</span><span>)</span> <span>else</span> <span>None</span>
        <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-offSlr'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span># https://regex101.com/r/lap8lr/1 </span>        <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
        <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-standardPriceDemoted'</span><span>)</span> <span>else</span> <span>None</span>
        <span>shipping_text</span> <span>=</span> <span>raw_shipping</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_shipping</span> <span>else</span> <span>''</span>
        <span>shipping</span> <span>=</span> <span>shipping_text</span> <span>if</span> <span>'shipping'</span> <span>in</span> <span>shipping_text</span> <span>else</span> <span>None</span>
        <span>rating_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'aria-label'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
        <span># https://regex101.com/r/D2YTRI/1 </span>        <span>rating</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d.\d|\d]+'</span><span>,</span> <span>rating_text</span><span>).</span><span>group</span><span>(</span><span>0</span><span>))</span> <span>if</span> <span>rating_text</span> <span>else</span> <span>None</span>
        <span>reviews_text</span> <span>=</span> <span>raw_rating</span><span>.</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>raw_rating</span> <span>else</span> <span>None</span>
        <span>reviews</span> <span>=</span> <span>reviews_text</span><span>.</span><span>replace</span><span>(</span><span>'('</span><span>,</span> <span>''</span><span>).</span><span>replace</span><span>(</span><span>')'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>reviews_text</span> <span>else</span> <span>None</span>
        <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.cico img'</span><span>)</span> <span>else</span> <span>None</span>

        <span>data</span><span>.</span><span>append</span><span>({</span>
            <span>'position'</span><span>:</span> <span>position</span><span>,</span>
            <span>'title'</span><span>:</span> <span>title</span><span>,</span>
            <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
            <span>'brand'</span><span>:</span> <span>brand</span><span>,</span>
            <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
            <span>'price'</span><span>:</span> <span>price</span><span>,</span>
            <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
            <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
            <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
            <span>'rating'</span><span>:</span> <span>rating</span><span>,</span>
            <span>'reviews'</span><span>:</span> <span>reviews</span><span>,</span>
            <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
        <span>})</span>

    <span>return</span> <span>data</span>
def get_inline_shopping_results(parser: LexborHTMLParser, product_counter: int) -> list:
    data = []

    for position, product in enumerate(parser.root.css('.br-gOffCard'), start=product_counter):
        raw_title = product.css_first('.br-offTtl span')
        raw_shipping = product.css_first('.br-offDec')
        raw_rating = product.css_first('.tags > span')

        title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-offTtl').text().strip()
        product_link = product.css_first('.br-offLink').attributes.get('href', '')
        brand = product.css_first('.br-offSecLbl').text().strip() if product.css_first('.br-offSecLbl') else None
        seller = product.css_first('.br-offSlr').text().strip()
        price = product.css_first('.br-price').text().strip()
        # https://regex101.com/r/lap8lr/1         extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
        old_price = product.css_first('.br-standardPriceDemoted').text().strip() if product.css_first('.br-standardPriceDemoted') else None
        shipping_text = raw_shipping.text().strip() if raw_shipping else ''
        shipping = shipping_text if 'shipping' in shipping_text else None
        rating_text = raw_rating.attributes.get('aria-label', '') if raw_rating else None
        # https://regex101.com/r/D2YTRI/1         rating = float(re.search(r'[\d.\d|\d]+', rating_text).group(0)) if rating_text else None
        reviews_text = raw_rating.text().strip() if raw_rating else None
        reviews = reviews_text.replace('(', '').replace(')', '') if reviews_text else None
        thumbnail = product.css_first('.cico img').attributes.get('src', None) if product.css_first('.cico img') else None

        data.append({
            'position': position,
            'title': title,
            'product_link': product_link,
            'brand': brand,
            'seller': seller,
            'price': price,
            'extracted_price': extracted_price,
            'old_price': old_price,
            'shipping': shipping,
            'rating': rating,
            'reviews': reviews,
            'thumbnail': thumbnail
        })

    return data

Enter fullscreen mode Exit fullscreen mode

Get shopping results

The function takes a parser and a product_counter. Returns a list with the retrieved data.

The data list is declared to which the extracted data will be added:

<span>data</span> <span>=</span> <span>[]</span>
<span>data</span> <span>=</span> <span>[]</span>
data = []

Enter fullscreen mode Exit fullscreen mode

In the previous function, the data extraction process was described in detail. In this function, the process is very similar except for different data and correspondingly different selectors:

<span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-fullCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
    <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title span'</span><span>)</span>
    <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
    <span>product_link</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-titlelink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
    <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-seller'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
    <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.pd-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
    <span># https://regex101.com/r/lap8lr/1 </span>    <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
    <span>sale</span> <span>=</span> <span>True</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-saletag'</span><span>)</span> <span>else</span> <span>False</span>
    <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>)</span> <span>else</span> <span>None</span>
    <span>shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>)</span> <span>else</span> <span>None</span>
    <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>)</span> <span>else</span> <span>None</span>
<span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-fullCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
    <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title span'</span><span>)</span>

    <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
    <span>product_link</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-titlelink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
    <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-seller'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
    <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.pd-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
    <span># https://regex101.com/r/lap8lr/1 </span>    <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
    <span>sale</span> <span>=</span> <span>True</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-saletag'</span><span>)</span> <span>else</span> <span>False</span>
    <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>)</span> <span>else</span> <span>None</span>
    <span>shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>)</span> <span>else</span> <span>None</span>
    <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>)</span> <span>else</span> <span>None</span>
for position, product in enumerate(parser.root.css('.br-fullCard'), start=product_counter):
    raw_title = product.css_first('.br-title span')

    title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-title').text().strip()
    product_link = f"https://www.bing.com{product.css_first('.br-titlelink').attributes.get('href', '')}"
    seller = product.css_first('.br-seller').text().strip()
    price = product.css_first('.pd-price').text().strip()
    # https://regex101.com/r/lap8lr/1     extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
    sale = True if product.css_first('.br-saletag') else False
    old_price = product.css_first('.br-pdOfferPrice').text().strip() if product.css_first('.br-pdOfferPrice') else None
    shipping = product.css_first('.br-decoSlot').text().strip() if product.css_first('.br-decoSlot') else None
    thumbnail = product.css_first('.br-pdMainImg img').attributes.get('src', None) if product.css_first('.br-pdMainImg img') else None

Enter fullscreen mode Exit fullscreen mode

At the end of each iteration, product data is added to the data list:

<span>data</span><span>.</span><span>append</span><span>({</span>
    <span>'position'</span><span>:</span> <span>position</span><span>,</span>
    <span>'title'</span><span>:</span> <span>title</span><span>,</span>
    <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
    <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
    <span>'price'</span><span>:</span> <span>price</span><span>,</span>
    <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
    <span>'sale'</span><span>:</span> <span>sale</span><span>,</span>
    <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
    <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
    <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
<span>})</span>
<span>data</span><span>.</span><span>append</span><span>({</span>
    <span>'position'</span><span>:</span> <span>position</span><span>,</span>
    <span>'title'</span><span>:</span> <span>title</span><span>,</span>
    <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
    <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
    <span>'price'</span><span>:</span> <span>price</span><span>,</span>
    <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
    <span>'sale'</span><span>:</span> <span>sale</span><span>,</span>
    <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
    <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
    <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
<span>})</span>
data.append({
    'position': position,
    'title': title,
    'product_link': product_link,
    'seller': seller,
    'price': price,
    'extracted_price': extracted_price,
    'sale': sale,
    'old_price': old_price,
    'shipping': shipping,
    'thumbnail': thumbnail
})

Enter fullscreen mode Exit fullscreen mode

After all the operations are done, return the data list:

<span>return</span> <span>data</span>
<span>return</span> <span>data</span>
return data

Enter fullscreen mode Exit fullscreen mode

The function looks like this:

<span>def</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>:</span> <span>LexborHTMLParser</span><span>,</span> <span>product_counter</span><span>:</span> <span>int</span><span>)</span> <span>-></span> <span>list</span><span>:</span>
    <span>data</span> <span>=</span> <span>[]</span>
    <span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-fullCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
        <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title span'</span><span>)</span>
        <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>product_link</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-titlelink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
        <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-seller'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.pd-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span># https://regex101.com/r/lap8lr/1 </span>        <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
        <span>sale</span> <span>=</span> <span>True</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-saletag'</span><span>)</span> <span>else</span> <span>False</span>
        <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>)</span> <span>else</span> <span>None</span>
        <span>shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>)</span> <span>else</span> <span>None</span>
        <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>)</span> <span>else</span> <span>None</span>
        <span>data</span><span>.</span><span>append</span><span>({</span>
            <span>'position'</span><span>:</span> <span>position</span><span>,</span>
            <span>'title'</span><span>:</span> <span>title</span><span>,</span>
            <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
            <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
            <span>'price'</span><span>:</span> <span>price</span><span>,</span>
            <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
            <span>'sale'</span><span>:</span> <span>sale</span><span>,</span>
            <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
            <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
            <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
        <span>})</span>
    <span>return</span> <span>data</span>
<span>def</span> <span>get_shopping_results</span><span>(</span><span>parser</span><span>:</span> <span>LexborHTMLParser</span><span>,</span> <span>product_counter</span><span>:</span> <span>int</span><span>)</span> <span>-></span> <span>list</span><span>:</span>
    <span>data</span> <span>=</span> <span>[]</span>

    <span>for</span> <span>position</span><span>,</span> <span>product</span> <span>in</span> <span>enumerate</span><span>(</span><span>parser</span><span>.</span><span>root</span><span>.</span><span>css</span><span>(</span><span>'.br-fullCard'</span><span>),</span> <span>start</span><span>=</span><span>product_counter</span><span>):</span>
        <span>raw_title</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title span'</span><span>)</span>

        <span>title</span> <span>=</span> <span>raw_title</span><span>.</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'title'</span><span>,</span> <span>''</span><span>)</span> <span>if</span> <span>raw_title</span> <span>else</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-title'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>product_link</span> <span>=</span> <span>f</span><span>"https://www.bing.com</span><span>{</span><span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-titlelink'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'href'</span><span>,</span> <span>''</span><span>)</span><span>}</span><span>"</span>
        <span>seller</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-seller'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span>price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.pd-price'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span>
        <span># https://regex101.com/r/lap8lr/1 </span>        <span>extracted_price</span> <span>=</span> <span>float</span><span>(</span><span>re</span><span>.</span><span>search</span><span>(</span><span>r</span><span>'[\d|,|.]+'</span><span>,</span> <span>price</span><span>).</span><span>group</span><span>().</span><span>replace</span><span>(</span><span>','</span><span>,</span> <span>''</span><span>))</span>
        <span>sale</span> <span>=</span> <span>True</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-saletag'</span><span>)</span> <span>else</span> <span>False</span>
        <span>old_price</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdOfferPrice'</span><span>)</span> <span>else</span> <span>None</span>
        <span>shipping</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>).</span><span>text</span><span>().</span><span>strip</span><span>()</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-decoSlot'</span><span>)</span> <span>else</span> <span>None</span>
        <span>thumbnail</span> <span>=</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>).</span><span>attributes</span><span>.</span><span>get</span><span>(</span><span>'src'</span><span>,</span> <span>None</span><span>)</span> <span>if</span> <span>product</span><span>.</span><span>css_first</span><span>(</span><span>'.br-pdMainImg img'</span><span>)</span> <span>else</span> <span>None</span>

        <span>data</span><span>.</span><span>append</span><span>({</span>
            <span>'position'</span><span>:</span> <span>position</span><span>,</span>
            <span>'title'</span><span>:</span> <span>title</span><span>,</span>
            <span>'product_link'</span><span>:</span> <span>product_link</span><span>,</span>
            <span>'seller'</span><span>:</span> <span>seller</span><span>,</span>
            <span>'price'</span><span>:</span> <span>price</span><span>,</span>
            <span>'extracted_price'</span><span>:</span> <span>extracted_price</span><span>,</span>
            <span>'sale'</span><span>:</span> <span>sale</span><span>,</span>
            <span>'old_price'</span><span>:</span> <span>old_price</span><span>,</span>
            <span>'shipping'</span><span>:</span> <span>shipping</span><span>,</span>
            <span>'thumbnail'</span><span>:</span> <span>thumbnail</span>
        <span>})</span>

    <span>return</span> <span>data</span>
def get_shopping_results(parser: LexborHTMLParser, product_counter: int) -> list:
    data = []

    for position, product in enumerate(parser.root.css('.br-fullCard'), start=product_counter):
        raw_title = product.css_first('.br-title span')

        title = raw_title.attributes.get('title', '') if raw_title else product.css_first('.br-title').text().strip()
        product_link = f"https://www.bing.com{product.css_first('.br-titlelink').attributes.get('href', '')}"
        seller = product.css_first('.br-seller').text().strip()
        price = product.css_first('.pd-price').text().strip()
        # https://regex101.com/r/lap8lr/1         extracted_price = float(re.search(r'[\d|,|.]+', price).group().replace(',', ''))
        sale = True if product.css_first('.br-saletag') else False
        old_price = product.css_first('.br-pdOfferPrice').text().strip() if product.css_first('.br-pdOfferPrice') else None
        shipping = product.css_first('.br-decoSlot').text().strip() if product.css_first('.br-decoSlot') else None
        thumbnail = product.css_first('.br-pdMainImg img').attributes.get('src', None) if product.css_first('.br-pdMainImg img') else None

        data.append({
            'position': position,
            'title': title,
            'product_link': product_link,
            'seller': seller,
            'price': price,
            'extracted_price': extracted_price,
            'sale': sale,
            'old_price': old_price,
            'shipping': shipping,
            'thumbnail': thumbnail
        })

    return data

Enter fullscreen mode Exit fullscreen mode

Output

<span>{</span><span> </span><span>"inline_shopping_results"</span><span>:</span><span> </span><span>[</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>1</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Sony Playstation 4 Dual Shock 4 Controller"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/aclick?ld=e8y8UnkNEA5yLnmFwFXVJsIDVUCUxGDjCRfBB7hezInXrQF_5sOcYbnlwQ7FroV_Zn5FefQcj7dQqTSlvA3lj2I21y0MviXMAVnyW-3FkoUoi16_LPCXsLblfhhQ2D_DBXPc7yCF56HaNeXUDrxLymGBGLDEWL241igNH5h1ZNrEBK3Hy1&u=aHR0cHMlM2ElMmYlMmZ3d3cuYW1hem9uLmNvbSUyZlNvbnktUGxheVN0YXRpb24tRHVhbHNob2NrLVdpcmVsZXNzLUNvbnRyb2xsZXItQmxhY2slMmZkcCUyZkIwMEQ4Mlo0WU8lMmZyZWYlM2Rhc2NfZGZfQjAwRDgyWjRZTyUzZnRhZyUzZGJpbmdzaG9wcGluZ2EtMjAlMjZsaW5rQ29kZSUzZGRmMCUyNmh2YWRpZCUzZDc5OTg5NTk3MTQ0NTE1JTI2aHZuZXR3JTNkbyUyNmh2cW10JTNkZSUyNmh2Ym10JTNkYmUlMjZodmRldiUzZGMlMjZodmxvY2ludCUzZCUyNmh2bG9jcGh5JTNkJTI2aHZ0YXJnaWQlM2RwbGEtNDU4MzU4OTExODI4NTM1MCUyNnBzYyUzZDE&rlid=044e0ea8d1d91e49a33a6a95b021d2b3"</span><span>,</span><span> </span><span>"brand"</span><span>:</span><span> </span><span>"580+ viewed"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Amazon.com"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$56.00"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>56.0</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"rating"</span><span>:</span><span> </span><span>4.5</span><span>,</span><span> </span><span>"reviews"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.javWvhvskVjF3A474C474&w=140&h=140&pid=21.1"</span><span> </span><span>},</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>2</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Wireless Controller For PS4/Slim/Pro, Berry Blue"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/aclick?ld=e8K5Qjv1rx7b91q1TGu2kIHDVUCUzljX6PIJ1_Ag71Iy5QZiGWnbe49xj4OnlDYYwXIZtyHeZHN66fGgfhttXqkZjtRPPAG3TZF7rGH0z6XerddgQBQx_dfYoJlYMoPPnC1ujLoNsFoUL6VTMyu4Ln14u3EoPqCJV8wSvMZ7FHfLWcEayH&u=aHR0cHMlM2ElMmYlMmZjbGlja3NlcnZlLmRhcnRzZWFyY2gubmV0JTJmbGluayUyZmNsaWNrJTNmJTI2JTI2ZHNfZV9hZGlkJTNkNzUzMTY2OTg4MTk5MTglMjZkc19lX3RhcmdldF9pZCUzZHBsYS00NTc4OTE2MjM5MDc0Njc5JTI2ZHNfZV9wcm9kdWN0X2dyb3VwX2lkJTNkNDU3ODkxNjIzOTA3NDY3OSUyNmRzX2VfcHJvZHVjdF9pZCUzZDEzOTY2MDA3OV8xMDAwMTEzNTk4OCUyNmRzX2VfcHJvZHVjdF9jb3VudHJ5JTNkVVMlMjZkc19lX3Byb2R1Y3RfbGFuZ3VhZ2UlM2RFTiUyNmRzX2VfcHJvZHVjdF9jaGFubmVsJTNkT25saW5lJTI2ZHNfdXJsX3YlM2QyJTI2ZHNfZGVzdF91cmwlM2RodHRwcyUzYSUyZiUyZnd3dy53YWxtYXJ0LmNvbSUyZmlwJTJmV2lyZWxlc3MtQ29udHJvbGxlci1mb3ItUFM0LVNsaW0tUHJvLUJlcnJ5LUJsdWUlMmYxMzk2NjAwNzklM2Z3bWxzcGFydG5lciUzZHdscGElMjZzZWxlY3RlZFNlbGxlcklkJTNkMTAxMTE2NjI2JTI2YWRpZCUzZDIyMjIyMjIyMjIyNTc2ODA1MzgwJTI2d21sc3BhcnRuZXIlM2R3bXRsYWJzJTI2d2wwJTNkZSUyNndsMSUzZG8lMjZ3bDIlM2RjJTI2d2wzJTNkNzUzMTY2OTg4MTk5MTglMjZ3bDQlM2RwbGEtNDU3ODkxNjIzOTA3NDY3OSUyNndsNSUzZCUyNndsNiUzZCUyNndsNyUzZCUyNndsMTAlM2RXYWxtYXJ0JTI2d2wxMSUzZE9ubGluZSUyNndsMTIlM2QxMzk2NjAwNzlfMTAwMDExMzU5ODglMjZ3bDE0JTNkcHM0JTI1MjBjb250cm9sbGVyJTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2Q1NjkxZjc3M2I3ODQxMTQ2ZTVhYjkzNGVkNjdjNTQ0OSUyNmdjbHNyYyUzZDNwLmRzJTI2bXNjbGtpZCUzZDU2OTFmNzczYjc4NDExNDZlNWFiOTM0ZWQ2N2M1NDQ5&rlid=5691f773b7841146e5ab934ed67c5449"</span><span>,</span><span> </span><span>"brand"</span><span>:</span><span> </span><span>"Brand: SPBPQY"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Walmart"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$19.99"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>19.99</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>"Free shipping"</span><span>,</span><span> </span><span>"rating"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"reviews"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.AjeAqUx49FCBaQ474C474&w=140&h=140&pid=21.1"</span><span> </span><span>},</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>3</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"YCCTEAM Wireless Ps4 Controller ,Wireless Game Controller Compatible With Playstation 4/Slim/Pro, Built-In 1000Mah Battery With Turbo/Dual Vibration"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/aclick?ld=e8XOuCV1B-F1vb3cgxnxYFWjVUCUxeNZ_G0jSncEe8IzWF8UzLDZihTUvXFtS_U7DFDfJNV_L4aBCyLLwDxIW3jnYGUaW0Q0C4MhhYByxkPdaXmHEexgmFgmfKA1EITFmoVR4_Yla_FXAV-Nmx6yJiQmPLtRwNLSr6eWXTZE07O-ZjlAPX&u=aHR0cHMlM2ElMmYlMmZ3d3cuYW1hem9uLmNvbSUyZkNvbnRyb2xsZXItQ2hhcmdpbmctWUNDVEVBTS1JbmRpY2F0b3ItSm95c3RpY2tzJTJmZHAlMmZCMDdXN0gyMVZWJTJmcmVmJTNkYXNjX2RmX0IwN1c3SDIxVlYlM2Z0YWclM2RiaW5nc2hvcHBpbmdhLTIwJTI2bGlua0NvZGUlM2RkZjAlMjZodmFkaWQlM2Q3OTg1MjEyNDE1MTUzNSUyNmh2bmV0dyUzZG8lMjZodnFtdCUzZGUlMjZodmJtdCUzZGJlJTI2aHZkZXYlM2RjJTI2aHZsb2NpbnQlM2QlMjZodmxvY3BoeSUzZCUyNmh2dGFyZ2lkJTNkcGxhLTQ1ODM0NTE2NjkzOTg1NzklMjZwc2MlM2Qx&rlid=15ad8be3c1af10f584ffefdea057bbd8"</span><span>,</span><span> </span><span>"brand"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Amazon.com"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$19.99"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>19.99</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"rating"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"reviews"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.NsyWoG%2bGBv4G3A474C474&w=140&h=140&pid=21.1"</span><span> </span><span>},</span><span> </span><span>...</span><span> </span><span>other</span><span> </span><span>inline</span><span> </span><span>shopping</span><span> </span><span>results</span><span> </span><span>],</span><span> </span><span>"shopping_results"</span><span>:</span><span> </span><span>[</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>1</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Sony PS4 Dualshock 4 Wireless Controller - Midnight Blue"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IKQSlWYtjMJCH45YKbCW1INZT5%22+gIdHash%3a%221034604334%22+gGlobalOfferIds%3a%2297596786905%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IKQSlWYtjMJCH45YKbCW1INZT5%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Walmart"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$64.00"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>64.0</span><span>,</span><span> </span><span>"sale"</span><span>:</span><span> </span><span>false</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>"Free shipping"</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.5KY3rYxghNq1ng474C474&w=272&h=272&o=5&pid=21.1"</span><span> </span><span>},</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>2</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Sony Dual Shock 4 Bluetooth Controller For PS4 - Black"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IjzPcGXIjh5kEfHzF4DdKkdm7O%22+gIdHash%3a%22503285226%22+gGlobalOfferIds%3a%2213378540660%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IjzPcGXIjh5kEfHzF4DdKkdm7O%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Tech Import World"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$51.92"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>51.92</span><span>,</span><span> </span><span>"sale"</span><span>:</span><span> </span><span>false</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.2zrQISY7yRggYA474C474&w=272&h=272&o=5&pid=21.1"</span><span> </span><span>},</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>3</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Sony PS4 Dualshock 4 Wireless Controller - Green Camouflage"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IsUwpe7StYMy9zWBEooypnccb6%22+gIdHash%3a%221451490157%22+gGlobalOfferIds%3a%2285554586355%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IsUwpe7StYMy9zWBEooypnccb6%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Walmart"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$64.00"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>64.0</span><span>,</span><span> </span><span>"sale"</span><span>:</span><span> </span><span>false</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>"Free shipping"</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.bmIHvaU977kYvw474C474&w=272&h=272&o=5&pid=21.1"</span><span> </span><span>},</span><span> </span><span>...</span><span> </span><span>other</span><span> </span><span>shopping</span><span> </span><span>results</span><span> </span><span>]</span><span> </span><span>}</span><span> </span>
<span>{</span><span> </span><span>"inline_shopping_results"</span><span>:</span><span> </span><span>[</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>1</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Sony Playstation 4 Dual Shock 4 Controller"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/aclick?ld=e8y8UnkNEA5yLnmFwFXVJsIDVUCUxGDjCRfBB7hezInXrQF_5sOcYbnlwQ7FroV_Zn5FefQcj7dQqTSlvA3lj2I21y0MviXMAVnyW-3FkoUoi16_LPCXsLblfhhQ2D_DBXPc7yCF56HaNeXUDrxLymGBGLDEWL241igNH5h1ZNrEBK3Hy1&u=aHR0cHMlM2ElMmYlMmZ3d3cuYW1hem9uLmNvbSUyZlNvbnktUGxheVN0YXRpb24tRHVhbHNob2NrLVdpcmVsZXNzLUNvbnRyb2xsZXItQmxhY2slMmZkcCUyZkIwMEQ4Mlo0WU8lMmZyZWYlM2Rhc2NfZGZfQjAwRDgyWjRZTyUzZnRhZyUzZGJpbmdzaG9wcGluZ2EtMjAlMjZsaW5rQ29kZSUzZGRmMCUyNmh2YWRpZCUzZDc5OTg5NTk3MTQ0NTE1JTI2aHZuZXR3JTNkbyUyNmh2cW10JTNkZSUyNmh2Ym10JTNkYmUlMjZodmRldiUzZGMlMjZodmxvY2ludCUzZCUyNmh2bG9jcGh5JTNkJTI2aHZ0YXJnaWQlM2RwbGEtNDU4MzU4OTExODI4NTM1MCUyNnBzYyUzZDE&rlid=044e0ea8d1d91e49a33a6a95b021d2b3"</span><span>,</span><span> </span><span>"brand"</span><span>:</span><span> </span><span>"580+ viewed"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Amazon.com"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$56.00"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>56.0</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"rating"</span><span>:</span><span> </span><span>4.5</span><span>,</span><span> </span><span>"reviews"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.javWvhvskVjF3A474C474&w=140&h=140&pid=21.1"</span><span> </span><span>},</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>2</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Wireless Controller For PS4/Slim/Pro, Berry Blue"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/aclick?ld=e8K5Qjv1rx7b91q1TGu2kIHDVUCUzljX6PIJ1_Ag71Iy5QZiGWnbe49xj4OnlDYYwXIZtyHeZHN66fGgfhttXqkZjtRPPAG3TZF7rGH0z6XerddgQBQx_dfYoJlYMoPPnC1ujLoNsFoUL6VTMyu4Ln14u3EoPqCJV8wSvMZ7FHfLWcEayH&u=aHR0cHMlM2ElMmYlMmZjbGlja3NlcnZlLmRhcnRzZWFyY2gubmV0JTJmbGluayUyZmNsaWNrJTNmJTI2JTI2ZHNfZV9hZGlkJTNkNzUzMTY2OTg4MTk5MTglMjZkc19lX3RhcmdldF9pZCUzZHBsYS00NTc4OTE2MjM5MDc0Njc5JTI2ZHNfZV9wcm9kdWN0X2dyb3VwX2lkJTNkNDU3ODkxNjIzOTA3NDY3OSUyNmRzX2VfcHJvZHVjdF9pZCUzZDEzOTY2MDA3OV8xMDAwMTEzNTk4OCUyNmRzX2VfcHJvZHVjdF9jb3VudHJ5JTNkVVMlMjZkc19lX3Byb2R1Y3RfbGFuZ3VhZ2UlM2RFTiUyNmRzX2VfcHJvZHVjdF9jaGFubmVsJTNkT25saW5lJTI2ZHNfdXJsX3YlM2QyJTI2ZHNfZGVzdF91cmwlM2RodHRwcyUzYSUyZiUyZnd3dy53YWxtYXJ0LmNvbSUyZmlwJTJmV2lyZWxlc3MtQ29udHJvbGxlci1mb3ItUFM0LVNsaW0tUHJvLUJlcnJ5LUJsdWUlMmYxMzk2NjAwNzklM2Z3bWxzcGFydG5lciUzZHdscGElMjZzZWxlY3RlZFNlbGxlcklkJTNkMTAxMTE2NjI2JTI2YWRpZCUzZDIyMjIyMjIyMjIyNTc2ODA1MzgwJTI2d21sc3BhcnRuZXIlM2R3bXRsYWJzJTI2d2wwJTNkZSUyNndsMSUzZG8lMjZ3bDIlM2RjJTI2d2wzJTNkNzUzMTY2OTg4MTk5MTglMjZ3bDQlM2RwbGEtNDU3ODkxNjIzOTA3NDY3OSUyNndsNSUzZCUyNndsNiUzZCUyNndsNyUzZCUyNndsMTAlM2RXYWxtYXJ0JTI2d2wxMSUzZE9ubGluZSUyNndsMTIlM2QxMzk2NjAwNzlfMTAwMDExMzU5ODglMjZ3bDE0JTNkcHM0JTI1MjBjb250cm9sbGVyJTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2Q1NjkxZjc3M2I3ODQxMTQ2ZTVhYjkzNGVkNjdjNTQ0OSUyNmdjbHNyYyUzZDNwLmRzJTI2bXNjbGtpZCUzZDU2OTFmNzczYjc4NDExNDZlNWFiOTM0ZWQ2N2M1NDQ5&rlid=5691f773b7841146e5ab934ed67c5449"</span><span>,</span><span> </span><span>"brand"</span><span>:</span><span> </span><span>"Brand: SPBPQY"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Walmart"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$19.99"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>19.99</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>"Free shipping"</span><span>,</span><span> </span><span>"rating"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"reviews"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.AjeAqUx49FCBaQ474C474&w=140&h=140&pid=21.1"</span><span> </span><span>},</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>3</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"YCCTEAM Wireless Ps4 Controller ,Wireless Game Controller Compatible With Playstation 4/Slim/Pro, Built-In 1000Mah Battery With Turbo/Dual Vibration"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/aclick?ld=e8XOuCV1B-F1vb3cgxnxYFWjVUCUxeNZ_G0jSncEe8IzWF8UzLDZihTUvXFtS_U7DFDfJNV_L4aBCyLLwDxIW3jnYGUaW0Q0C4MhhYByxkPdaXmHEexgmFgmfKA1EITFmoVR4_Yla_FXAV-Nmx6yJiQmPLtRwNLSr6eWXTZE07O-ZjlAPX&u=aHR0cHMlM2ElMmYlMmZ3d3cuYW1hem9uLmNvbSUyZkNvbnRyb2xsZXItQ2hhcmdpbmctWUNDVEVBTS1JbmRpY2F0b3ItSm95c3RpY2tzJTJmZHAlMmZCMDdXN0gyMVZWJTJmcmVmJTNkYXNjX2RmX0IwN1c3SDIxVlYlM2Z0YWclM2RiaW5nc2hvcHBpbmdhLTIwJTI2bGlua0NvZGUlM2RkZjAlMjZodmFkaWQlM2Q3OTg1MjEyNDE1MTUzNSUyNmh2bmV0dyUzZG8lMjZodnFtdCUzZGUlMjZodmJtdCUzZGJlJTI2aHZkZXYlM2RjJTI2aHZsb2NpbnQlM2QlMjZodmxvY3BoeSUzZCUyNmh2dGFyZ2lkJTNkcGxhLTQ1ODM0NTE2NjkzOTg1NzklMjZwc2MlM2Qx&rlid=15ad8be3c1af10f584ffefdea057bbd8"</span><span>,</span><span> </span><span>"brand"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Amazon.com"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$19.99"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>19.99</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"rating"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"reviews"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.NsyWoG%2bGBv4G3A474C474&w=140&h=140&pid=21.1"</span><span> </span><span>},</span><span> </span><span>...</span><span> </span><span>other</span><span> </span><span>inline</span><span> </span><span>shopping</span><span> </span><span>results</span><span> </span><span>],</span><span> </span><span>"shopping_results"</span><span>:</span><span> </span><span>[</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>1</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Sony PS4 Dualshock 4 Wireless Controller - Midnight Blue"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IKQSlWYtjMJCH45YKbCW1INZT5%22+gIdHash%3a%221034604334%22+gGlobalOfferIds%3a%2297596786905%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IKQSlWYtjMJCH45YKbCW1INZT5%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Walmart"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$64.00"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>64.0</span><span>,</span><span> </span><span>"sale"</span><span>:</span><span> </span><span>false</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>"Free shipping"</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.5KY3rYxghNq1ng474C474&w=272&h=272&o=5&pid=21.1"</span><span> </span><span>},</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>2</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Sony Dual Shock 4 Bluetooth Controller For PS4 - Black"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IjzPcGXIjh5kEfHzF4DdKkdm7O%22+gIdHash%3a%22503285226%22+gGlobalOfferIds%3a%2213378540660%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IjzPcGXIjh5kEfHzF4DdKkdm7O%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Tech Import World"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$51.92"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>51.92</span><span>,</span><span> </span><span>"sale"</span><span>:</span><span> </span><span>false</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.2zrQISY7yRggYA474C474&w=272&h=272&o=5&pid=21.1"</span><span> </span><span>},</span><span> </span><span>{</span><span> </span><span>"position"</span><span>:</span><span> </span><span>3</span><span>,</span><span> </span><span>"title"</span><span>:</span><span> </span><span>"Sony PS4 Dualshock 4 Wireless Controller - Green Camouflage"</span><span>,</span><span> </span><span>"product_link"</span><span>:</span><span> </span><span>"https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IsUwpe7StYMy9zWBEooypnccb6%22+gIdHash%3a%221451490157%22+gGlobalOfferIds%3a%2285554586355%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IsUwpe7StYMy9zWBEooypnccb6%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true"</span><span>,</span><span> </span><span>"seller"</span><span>:</span><span> </span><span>"Walmart"</span><span>,</span><span> </span><span>"price"</span><span>:</span><span> </span><span>"$64.00"</span><span>,</span><span> </span><span>"extracted_price"</span><span>:</span><span> </span><span>64.0</span><span>,</span><span> </span><span>"sale"</span><span>:</span><span> </span><span>false</span><span>,</span><span> </span><span>"old_price"</span><span>:</span><span> </span><span>null</span><span>,</span><span> </span><span>"shipping"</span><span>:</span><span> </span><span>"Free shipping"</span><span>,</span><span> </span><span>"thumbnail"</span><span>:</span><span> </span><span>"https://th.bing.com/th?id=OP.bmIHvaU977kYvw474C474&w=272&h=272&o=5&pid=21.1"</span><span> </span><span>},</span><span> </span><span>...</span><span> </span><span>other</span><span> </span><span>shopping</span><span> </span><span>results</span><span> </span><span>]</span><span> </span><span>}</span><span> </span>
{ "inline_shopping_results": [ { "position": 1, "title": "Sony Playstation 4 Dual Shock 4 Controller", "product_link": "https://www.bing.com/aclick?ld=e8y8UnkNEA5yLnmFwFXVJsIDVUCUxGDjCRfBB7hezInXrQF_5sOcYbnlwQ7FroV_Zn5FefQcj7dQqTSlvA3lj2I21y0MviXMAVnyW-3FkoUoi16_LPCXsLblfhhQ2D_DBXPc7yCF56HaNeXUDrxLymGBGLDEWL241igNH5h1ZNrEBK3Hy1&u=aHR0cHMlM2ElMmYlMmZ3d3cuYW1hem9uLmNvbSUyZlNvbnktUGxheVN0YXRpb24tRHVhbHNob2NrLVdpcmVsZXNzLUNvbnRyb2xsZXItQmxhY2slMmZkcCUyZkIwMEQ4Mlo0WU8lMmZyZWYlM2Rhc2NfZGZfQjAwRDgyWjRZTyUzZnRhZyUzZGJpbmdzaG9wcGluZ2EtMjAlMjZsaW5rQ29kZSUzZGRmMCUyNmh2YWRpZCUzZDc5OTg5NTk3MTQ0NTE1JTI2aHZuZXR3JTNkbyUyNmh2cW10JTNkZSUyNmh2Ym10JTNkYmUlMjZodmRldiUzZGMlMjZodmxvY2ludCUzZCUyNmh2bG9jcGh5JTNkJTI2aHZ0YXJnaWQlM2RwbGEtNDU4MzU4OTExODI4NTM1MCUyNnBzYyUzZDE&rlid=044e0ea8d1d91e49a33a6a95b021d2b3", "brand": "580+ viewed", "seller": "Amazon.com", "price": "$56.00", "extracted_price": 56.0, "old_price": null, "shipping": null, "rating": 4.5, "reviews": null, "thumbnail": "https://th.bing.com/th?id=OP.javWvhvskVjF3A474C474&w=140&h=140&pid=21.1" }, { "position": 2, "title": "Wireless Controller For PS4/Slim/Pro, Berry Blue", "product_link": "https://www.bing.com/aclick?ld=e8K5Qjv1rx7b91q1TGu2kIHDVUCUzljX6PIJ1_Ag71Iy5QZiGWnbe49xj4OnlDYYwXIZtyHeZHN66fGgfhttXqkZjtRPPAG3TZF7rGH0z6XerddgQBQx_dfYoJlYMoPPnC1ujLoNsFoUL6VTMyu4Ln14u3EoPqCJV8wSvMZ7FHfLWcEayH&u=aHR0cHMlM2ElMmYlMmZjbGlja3NlcnZlLmRhcnRzZWFyY2gubmV0JTJmbGluayUyZmNsaWNrJTNmJTI2JTI2ZHNfZV9hZGlkJTNkNzUzMTY2OTg4MTk5MTglMjZkc19lX3RhcmdldF9pZCUzZHBsYS00NTc4OTE2MjM5MDc0Njc5JTI2ZHNfZV9wcm9kdWN0X2dyb3VwX2lkJTNkNDU3ODkxNjIzOTA3NDY3OSUyNmRzX2VfcHJvZHVjdF9pZCUzZDEzOTY2MDA3OV8xMDAwMTEzNTk4OCUyNmRzX2VfcHJvZHVjdF9jb3VudHJ5JTNkVVMlMjZkc19lX3Byb2R1Y3RfbGFuZ3VhZ2UlM2RFTiUyNmRzX2VfcHJvZHVjdF9jaGFubmVsJTNkT25saW5lJTI2ZHNfdXJsX3YlM2QyJTI2ZHNfZGVzdF91cmwlM2RodHRwcyUzYSUyZiUyZnd3dy53YWxtYXJ0LmNvbSUyZmlwJTJmV2lyZWxlc3MtQ29udHJvbGxlci1mb3ItUFM0LVNsaW0tUHJvLUJlcnJ5LUJsdWUlMmYxMzk2NjAwNzklM2Z3bWxzcGFydG5lciUzZHdscGElMjZzZWxlY3RlZFNlbGxlcklkJTNkMTAxMTE2NjI2JTI2YWRpZCUzZDIyMjIyMjIyMjIyNTc2ODA1MzgwJTI2d21sc3BhcnRuZXIlM2R3bXRsYWJzJTI2d2wwJTNkZSUyNndsMSUzZG8lMjZ3bDIlM2RjJTI2d2wzJTNkNzUzMTY2OTg4MTk5MTglMjZ3bDQlM2RwbGEtNDU3ODkxNjIzOTA3NDY3OSUyNndsNSUzZCUyNndsNiUzZCUyNndsNyUzZCUyNndsMTAlM2RXYWxtYXJ0JTI2d2wxMSUzZE9ubGluZSUyNndsMTIlM2QxMzk2NjAwNzlfMTAwMDExMzU5ODglMjZ3bDE0JTNkcHM0JTI1MjBjb250cm9sbGVyJTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2Q1NjkxZjc3M2I3ODQxMTQ2ZTVhYjkzNGVkNjdjNTQ0OSUyNmdjbHNyYyUzZDNwLmRzJTI2bXNjbGtpZCUzZDU2OTFmNzczYjc4NDExNDZlNWFiOTM0ZWQ2N2M1NDQ5&rlid=5691f773b7841146e5ab934ed67c5449", "brand": "Brand: SPBPQY", "seller": "Walmart", "price": "$19.99", "extracted_price": 19.99, "old_price": null, "shipping": "Free shipping", "rating": null, "reviews": null, "thumbnail": "https://th.bing.com/th?id=OP.AjeAqUx49FCBaQ474C474&w=140&h=140&pid=21.1" }, { "position": 3, "title": "YCCTEAM Wireless Ps4 Controller ,Wireless Game Controller Compatible With Playstation 4/Slim/Pro, Built-In 1000Mah Battery With Turbo/Dual Vibration", "product_link": "https://www.bing.com/aclick?ld=e8XOuCV1B-F1vb3cgxnxYFWjVUCUxeNZ_G0jSncEe8IzWF8UzLDZihTUvXFtS_U7DFDfJNV_L4aBCyLLwDxIW3jnYGUaW0Q0C4MhhYByxkPdaXmHEexgmFgmfKA1EITFmoVR4_Yla_FXAV-Nmx6yJiQmPLtRwNLSr6eWXTZE07O-ZjlAPX&u=aHR0cHMlM2ElMmYlMmZ3d3cuYW1hem9uLmNvbSUyZkNvbnRyb2xsZXItQ2hhcmdpbmctWUNDVEVBTS1JbmRpY2F0b3ItSm95c3RpY2tzJTJmZHAlMmZCMDdXN0gyMVZWJTJmcmVmJTNkYXNjX2RmX0IwN1c3SDIxVlYlM2Z0YWclM2RiaW5nc2hvcHBpbmdhLTIwJTI2bGlua0NvZGUlM2RkZjAlMjZodmFkaWQlM2Q3OTg1MjEyNDE1MTUzNSUyNmh2bmV0dyUzZG8lMjZodnFtdCUzZGUlMjZodmJtdCUzZGJlJTI2aHZkZXYlM2RjJTI2aHZsb2NpbnQlM2QlMjZodmxvY3BoeSUzZCUyNmh2dGFyZ2lkJTNkcGxhLTQ1ODM0NTE2NjkzOTg1NzklMjZwc2MlM2Qx&rlid=15ad8be3c1af10f584ffefdea057bbd8", "brand": null, "seller": "Amazon.com", "price": "$19.99", "extracted_price": 19.99, "old_price": null, "shipping": null, "rating": null, "reviews": null, "thumbnail": "https://th.bing.com/th?id=OP.NsyWoG%2bGBv4G3A474C474&w=140&h=140&pid=21.1" }, ... other inline shopping results ], "shopping_results": [ { "position": 1, "title": "Sony PS4 Dualshock 4 Wireless Controller - Midnight Blue", "product_link": "https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IKQSlWYtjMJCH45YKbCW1INZT5%22+gIdHash%3a%221034604334%22+gGlobalOfferIds%3a%2297596786905%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IKQSlWYtjMJCH45YKbCW1INZT5%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true", "seller": "Walmart", "price": "$64.00", "extracted_price": 64.0, "sale": false, "old_price": null, "shipping": "Free shipping", "thumbnail": "https://th.bing.com/th?id=OP.5KY3rYxghNq1ng474C474&w=272&h=272&o=5&pid=21.1" }, { "position": 2, "title": "Sony Dual Shock 4 Bluetooth Controller For PS4 - Black", "product_link": "https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IjzPcGXIjh5kEfHzF4DdKkdm7O%22+gIdHash%3a%22503285226%22+gGlobalOfferIds%3a%2213378540660%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IjzPcGXIjh5kEfHzF4DdKkdm7O%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true", "seller": "Tech Import World", "price": "$51.92", "extracted_price": 51.92, "sale": false, "old_price": null, "shipping": null, "thumbnail": "https://th.bing.com/th?id=OP.2zrQISY7yRggYA474C474&w=272&h=272&o=5&pid=21.1" }, { "position": 3, "title": "Sony PS4 Dualshock 4 Wireless Controller - Green Camouflage", "product_link": "https://www.bing.com/shop/productpage?q=ps4+controller&filters=scenario%3a%2217%22+gType%3a%223%22+gId%3a%22IsUwpe7StYMy9zWBEooypnccb6%22+gIdHash%3a%221451490157%22+gGlobalOfferIds%3a%2285554586355%22+AucContextGuid%3a%220%22+GroupEntityId%3a%22IsUwpe7StYMy9zWBEooypnccb6%22+NonSponsoredOffer%3a%22True%22&productpage=true&FORM=SHPPDP&browse=true", "seller": "Walmart", "price": "$64.00", "extracted_price": 64.0, "sale": false, "old_price": null, "shipping": "Free shipping", "thumbnail": "https://th.bing.com/th?id=OP.bmIHvaU977kYvw474C474&w=272&h=272&o=5&pid=21.1" }, ... other shopping results ] } 

Enter fullscreen mode Exit fullscreen mode

Links

• Code in the online IDE
• Requests
• Selectolax Lexbor module

Join us on Twitter | YouTube

Add a Feature Request or a Bug

Web Scraping (46 Part Series)

1 Scrape Google Shopping Tab with Python
2 Using Google Reverse Images API from SerpApi
… 42 more parts…
3 Scrape YouTube video page with Python
4 Scrape YouTube autocomplete results with Python
5 Scrape Google Realtime Search Trends with Python
6 Scrape Google Daily Search Trends with Python
7 Scrape Google Arts & Culture – Artists: All/A-Z/Time results with Python
8 Scrape Google Jobs organic results with Python
9 Scrape Google Play Games with Python
10 Scrape Google Play Apps with Python
11 Scrape Google Play Movies & TV with Python
12 Scrape Google Play Children (Kids) with Python
13 Scrape Google Play Books with Python
14 Scrape Brave Search Organic Results with Python
15 Scrape Brave News with Python
16 Scrape Brave Videos with Python
17 Scrape Brave Images with Python
18 Using Google Trends API from SerpApi
19 Using Google Jobs Listing Results API from SerpApi
20 Scrape Google Events Results with Python
21 Scrape Google Product Page with Python
22 Scrape Google Product Reviews Results with Python
23 Scrape Google Product Specs Results with Python
24 Scrape Google Product Online Sellers with Python
25 Using Google Product Local Sellers API from SerpApi
26 Using Google Maps Local Results API from SerpApi
27 Using Google Maps Place Results API from SerpApi
28 Using Google Maps Photos API from SerpApi
29 Using Google Maps Reviews API from SerpApi
30 Scrape Yelp Filters, Ad and Organic Results with Python
31 Using Yelp Reviews API from SerpApi with Python
32 Integrate The Home Depot Search Page Results Data with SerpApi and Python
33 How to Scrape Home Depot Product Data with SerpApi
34 Using Walmart Search Engine Results API from SerpApi
35 Using Walmart Product API from SerpApi
36 Scrape Google Lens with Python
37 SerpApi Changelog: December, 2022
38 Scraping Apple App Store Search with Python
39 Scraping Apple App Store Product Info And Reviews with Python
40 Scrape Google “Things to do” page with Python
41 Scrape Google Hotels with Python
42 Scraping Bing Organic Results using Python and SerpApi
43 How to Extract Bing News Data with SerpApi and Python
44 How to Extract Bing Images Data with SerpApi and Python
45 Scrape Google Flights with Python
46 Get Product Data from Bing Shopping with Python and SerpApi

原文链接：Get Product Data from Bing Shopping with Python and SerpApi