BeautifulSoup Cheat Sheet

Python Installtion


pip <span>install </span>beautifulsoup4
pip <span>install </span>beautifulsoup4
pip install beautifulsoup4

Enter fullscreen mode Exit fullscreen mode

Import Library


from bs4 import BeautifulSoup
from bs4 import BeautifulSoup
from bs4 import BeautifulSoup

Enter fullscreen mode Exit fullscreen mode

Creating a BeautifulSoup Object

Parse HTML string:


html <span>=</span> <span>"<p>Example paragraph</p>"</span>
soup <span>=</span> BeautifulSoup<span>(</span>html, <span>'html.parser'</span><span>)</span>
html <span>=</span> <span>"<p>Example paragraph</p>"</span>
soup <span>=</span> BeautifulSoup<span>(</span>html, <span>'html.parser'</span><span>)</span>
html = "<p>Example paragraph</p>"
soup = BeautifulSoup(html, 'html.parser')

Enter fullscreen mode Exit fullscreen mode

Parse from file


with open<span>(</span><span>"index.html"</span><span>)</span> as file:
  soup <span>=</span> BeautifulSoup<span>(</span>file, <span>'html.parser'</span><span>)</span>
with open<span>(</span><span>"index.html"</span><span>)</span> as file:
  soup <span>=</span> BeautifulSoup<span>(</span>file, <span>'html.parser'</span><span>)</span>
with open("index.html") as file:
  soup = BeautifulSoup(file, 'html.parser')

Enter fullscreen mode Exit fullscreen mode

BeautifulSoup Object Types

Tag

A Tag corresponds to an HTML or XML tag in the original document:


soup <span>=</span> BeautifulSoup<span>(</span><span>'<p>Hello World</p>'</span><span>)</span>
p_tag <span>=</span> soup.p
p_tag.name <span># 'p' <p></span>
p_tag.string <span># 'Hello World'</span>
soup <span>=</span> BeautifulSoup<span>(</span><span>'<p>Hello World</p>'</span><span>)</span>
p_tag <span>=</span> soup.p
p_tag.name <span># 'p' <p></span>
p_tag.string <span># 'Hello World'</span>
soup = BeautifulSoup('<p>Hello World</p>')
p_tag = soup.p
p_tag.name # 'p' <p>
p_tag.string # 'Hello World'

Enter fullscreen mode Exit fullscreen mode

NavigableString


soup <span>=</span> BeautifulSoup<span>(</span><span>'Hello World'</span><span>)</span>
text <span>=</span> soup.string
text <span># 'Hello World'</span>
<span>type</span><span>(</span>text<span>)</span> <span># bs4.element.NavigableString</span>
soup <span>=</span> BeautifulSoup<span>(</span><span>'Hello World'</span><span>)</span>
text <span>=</span> soup.string
text <span># 'Hello World'</span>
<span>type</span><span>(</span>text<span>)</span> <span># bs4.element.NavigableString</span>
soup = BeautifulSoup('Hello World')
text = soup.string
text # 'Hello World'
type(text) # bs4.element.NavigableString

Enter fullscreen mode Exit fullscreen mode

BeautifulSoup

The BeautifulSoup object represents the parsed document as a whole. It is the root of the tree:


soup <span>=</span> BeautifulSoup<span>(</span><span>'<html>...</html>'</span><span>)</span>
soup.name <span># '[document]'</span>
soup.head <span># <head> Tag element</span>
soup <span>=</span> BeautifulSoup<span>(</span><span>'<html>...</html>'</span><span>)</span>
soup.name <span># '[document]'</span>
soup.head <span># <head> Tag element</span>
soup = BeautifulSoup('<html>...</html>')
soup.name # '[document]'
soup.head # <head> Tag element

Enter fullscreen mode Exit fullscreen mode

Comment

Comments in HTML are also available as Comment objects:


<<span>!</span><span>--</span> This is a comment <span>--</span><span>></span>
<<span>!</span><span>--</span> This is a comment <span>--</span><span>></span>
<!-- This is a comment -->

Enter fullscreen mode Exit fullscreen mode


comment <span>=</span> soup.find<span>(</span><span>text</span><span>=</span>re.compile<span>(</span><span>'This is'</span><span>))</span>
<span>type</span><span>(</span>comment<span>)</span> <span># bs4.element.Comment</span>
comment <span>=</span> soup.find<span>(</span><span>text</span><span>=</span>re.compile<span>(</span><span>'This is'</span><span>))</span>
<span>type</span><span>(</span>comment<span>)</span> <span># bs4.element.Comment</span>
comment = soup.find(text=re.compile('This is'))
type(comment) # bs4.element.Comment

Enter fullscreen mode Exit fullscreen mode

Searching the Parse Tree

By Name

HTML:


<div>
  <p>Paragraph 1</p>
  <p>Paragraph 2</p>
</div>
<div>
  <p>Paragraph 1</p>
  <p>Paragraph 2</p>
</div>
<div>
  <p>Paragraph 1</p>
  <p>Paragraph 2</p>
</div>

Enter fullscreen mode Exit fullscreen mode


paragraphs <span>=</span> soup.find_all<span>(</span><span>'p'</span><span>)</span>
<span># <p>Paragraph 1</p>, <p>Paragraph 2</p></span>
paragraphs <span>=</span> soup.find_all<span>(</span><span>'p'</span><span>)</span>
<span># <p>Paragraph 1</p>, <p>Paragraph 2</p></span>
paragraphs = soup.find_all('p')
# <p>Paragraph 1</p>, <p>Paragraph 2</p>

Enter fullscreen mode Exit fullscreen mode

By Attributes

HTML:


<div <span>id</span><span>=</span><span>"content"</span><span>></span>
  <p>Paragraph 1</p>
</div>
<div <span>id</span><span>=</span><span>"content"</span><span>></span>
  <p>Paragraph 1</p>
</div>
<div id="content">
  <p>Paragraph 1</p>
</div>

Enter fullscreen mode Exit fullscreen mode

Python:


div <span>=</span> soup.find<span>(</span><span>id</span><span>=</span><span>"content"</span><span>)</span>
<span># <div id="content">...</div></span>
div <span>=</span> soup.find<span>(</span><span>id</span><span>=</span><span>"content"</span><span>)</span>
<span># <div id="content">...</div></span>
div = soup.find(id="content")
# <div id="content">...</div>

Enter fullscreen mode Exit fullscreen mode

Commonly Used `find` and `select` Methods in BeautifulSoup

By Text
HTML:


<p>This is some text</p>
<p>This is some text</p>
<p>This is some text</p>

Enter fullscreen mode Exit fullscreen mode

Python:


p <span>=</span> soup.find<span>(</span><span>text</span><span>=</span><span>"This is some text"</span><span>)</span>
<span># <p>This is some text</p></span>
p <span>=</span> soup.find<span>(</span><span>text</span><span>=</span><span>"This is some text"</span><span>)</span>
<span># <p>This is some text</p></span>
p = soup.find(text="This is some text")
# <p>This is some text</p>

Enter fullscreen mode Exit fullscreen mode

Searching with CSS Selectors
CSS selectors provide a very powerful way to search for elements within a parsed document.

Some examples of CSS selector syntax:

By Tag Name
Select all

tags:


soup.select<span>(</span><span>"p"</span><span>)</span>
soup.select<span>(</span><span>"p"</span><span>)</span>
soup.select("p")

Enter fullscreen mode Exit fullscreen mode

By ID
Select element with ID “main”:


soup.select<span>(</span><span>"#main"</span><span>)</span>
soup.select<span>(</span><span>"#main"</span><span>)</span>
soup.select("#main")

Enter fullscreen mode Exit fullscreen mode

By Class Name
Select elements with class “article”:


soup.select<span>(</span><span>".article"</span><span>)</span>
soup.select<span>(</span><span>".article"</span><span>)</span>
soup.select(".article")

Enter fullscreen mode Exit fullscreen mode

By Attribute
Select tags with a “data-category” attribute:


soup.select<span>(</span><span>"[data-category]"</span><span>)</span>
soup.select<span>(</span><span>"[data-category]"</span><span>)</span>
soup.select("[data-category]")

Enter fullscreen mode Exit fullscreen mode

Descendant Combinator
Select paragraphs inside divs:


soup.select<span>(</span><span>"div p"</span><span>)</span>
soup.select<span>(</span><span>"div p"</span><span>)</span>
soup.select("div p")

Enter fullscreen mode Exit fullscreen mode

Child Combinator
Select direct children paragraphs:


soup.select<span>(</span><span>"div > p"</span><span>)</span>
soup.select<span>(</span><span>"div > p"</span><span>)</span>
soup.select("div > p")

Enter fullscreen mode Exit fullscreen mode

Adjacent Sibling
Select h2 after h1:


soup.select<span>(</span><span>"h1 + h2"</span><span>)</span>
soup.select<span>(</span><span>"h1 + h2"</span><span>)</span>
soup.select("h1 + h2")

Enter fullscreen mode Exit fullscreen mode

General Sibling
Select h2 after any h1:


soup.select<span>(</span><span>"h1 ~ h2"</span><span>)</span>
soup.select<span>(</span><span>"h1 ~ h2"</span><span>)</span>
soup.select("h1 ~ h2")

Enter fullscreen mode Exit fullscreen mode

By Text
Select elements containing text:


soup.select<span>(</span><span>":contains('Some text')"</span><span>)</span>
soup.select<span>(</span><span>":contains('Some text')"</span><span>)</span>
soup.select(":contains('Some text')")

Enter fullscreen mode Exit fullscreen mode

By Attribute Value
Select input with type submit:


soup.select<span>(</span><span>"input[type='submit']"</span><span>)</span>
soup.select<span>(</span><span>"input[type='submit']"</span><span>)</span>
soup.select("input[type='submit']")

Enter fullscreen mode Exit fullscreen mode

Pseudo-classes
Select first paragraph:


soup.select<span>(</span><span>"p:first-of-type"</span><span>)</span>
soup.select<span>(</span><span>"p:first-of-type"</span><span>)</span>
soup.select("p:first-of-type")

Enter fullscreen mode Exit fullscreen mode

Chaining
Select first article paragraph:


soup.select<span>(</span><span>"article > p:nth-of-type(1)"</span><span>)</span>
soup.select<span>(</span><span>"article > p:nth-of-type(1)"</span><span>)</span>
soup.select("article > p:nth-of-type(1)")

Enter fullscreen mode Exit fullscreen mode

Accessing Data
HTML:


<p <span>class</span><span>=</span><span>"content"</span><span>></span>Some text</p>
<p <span>class</span><span>=</span><span>"content"</span><span>></span>Some text</p>
<p class="content">Some text</p>

Enter fullscreen mode Exit fullscreen mode

Python:


p <span>=</span> soup.find<span>(</span><span>'p'</span><span>)</span>
p.name <span># "p"</span>
p.attrs <span># {"class": "content"}</span>
p.string <span># "Some text"</span>
p <span>=</span> soup.find<span>(</span><span>'p'</span><span>)</span>
p.name <span># "p"</span>
p.attrs <span># {"class": "content"}</span>
p.string <span># "Some text"</span>
p = soup.find('p')
p.name # "p"
p.attrs # {"class": "content"}
p.string # "Some text"

Enter fullscreen mode Exit fullscreen mode

The Power of find_all()
The find_all() method is one of the most useful and versatile searching methods in BeautifulSoup.

Returns All Matches
find_all() will find and return a list of all matching elements:


all_paras <span>=</span> soup.find_all<span>(</span><span>'p'</span><span>)</span>
all_paras <span>=</span> soup.find_all<span>(</span><span>'p'</span><span>)</span>
all_paras = soup.find_all('p')

Enter fullscreen mode Exit fullscreen mode

This gives you all paragraphs on a page.

Flexible Queries
You can pass a wide range of queries to find_all():

Name – find_all(‘p’)
Attributes – find_all(‘a’, class_=’external’)
Text – find_all(text=re.compile(‘summary’))
Limit – find_all(‘p’, limit=2)
And more!
Useful Features
Some useful things you can do with find_all():

Get a count – len(soup.find_all(‘p’))
Iterate through results – for p in soup.find_all(‘p’):
Convert to text – [p.get_text() for p in soup.find_all(‘p’)]
Extract attributes – [a[‘href’] for a in soup.find_all(‘a’)]
Why It’s Useful
In summary, find_all() is useful because:
It returns all matching elements
It supports diverse and powerful queries
It enables easily extracting and processing result data
Whenever you need to get a collection of elements from a parsed document, find_all() will likely be your go-to tool.

Navigating Trees
Traverse up and sideways through related elements.

Modifying the Parse Tree
BeautifulSoup provides several methods for editing and modifying the parsed document tree.

HTML:


<p>Original text</p>
<p>Original text</p>
<p>Original text</p>

Enter fullscreen mode Exit fullscreen mode

Python:


p <span>=</span> soup.find<span>(</span><span>'p'</span><span>)</span>
p.string <span>=</span> <span>"New text"</span>
p <span>=</span> soup.find<span>(</span><span>'p'</span><span>)</span>
p.string <span>=</span> <span>"New text"</span>
p = soup.find('p')
p.string = "New text"

Enter fullscreen mode Exit fullscreen mode

Edit Tag Names
Change an existing tag name:


tag <span>=</span> soup.find<span>(</span><span>'span'</span><span>)</span>
tag.name <span>=</span> <span>'div'</span>
tag <span>=</span> soup.find<span>(</span><span>'span'</span><span>)</span>
tag.name <span>=</span> <span>'div'</span>
tag = soup.find('span')
tag.name = 'div'

Enter fullscreen mode Exit fullscreen mode

Edit Attributes
Add, modify or delete attributes of a tag:


tag[<span>'class'</span><span>]</span> <span>=</span> <span>'header'</span> <span># set attribute</span>
tag[<span>'id'</span><span>]</span> <span>=</span> <span>'main'</span>
del tag[<span>'class'</span><span>]</span> <span># delete attribute</span>
tag[<span>'class'</span><span>]</span> <span>=</span> <span>'header'</span> <span># set attribute</span>
tag[<span>'id'</span><span>]</span> <span>=</span> <span>'main'</span>
del tag[<span>'class'</span><span>]</span> <span># delete attribute</span>
tag['class'] = 'header' # set attribute
tag['id'] = 'main'
del tag['class'] # delete attribute

Enter fullscreen mode Exit fullscreen mode

Edit Text
Change text of a tag:


tag.string <span>=</span> <span>"New text"</span>
tag.string <span>=</span> <span>"New text"</span>
tag.string = "New text"

Enter fullscreen mode Exit fullscreen mode

Append text to a tag:


tag.append<span>(</span><span>"Additional text"</span><span>)</span>
tag.append<span>(</span><span>"Additional text"</span><span>)</span>
tag.append("Additional text")

Enter fullscreen mode Exit fullscreen mode

Insert Tags
Insert a new tag:


new_tag <span>=</span> soup.new_tag<span>(</span><span>"h1"</span><span>)</span>
tag.insert_before<span>(</span>new_tag<span>)</span>
new_tag <span>=</span> soup.new_tag<span>(</span><span>"h1"</span><span>)</span>
tag.insert_before<span>(</span>new_tag<span>)</span>
new_tag = soup.new_tag("h1")
tag.insert_before(new_tag)

Enter fullscreen mode Exit fullscreen mode

Delete Tags
Remove a tag entirely:


tag.extract<span>()</span>
tag.extract<span>()</span>
tag.extract()

Enter fullscreen mode Exit fullscreen mode

Wrap/Unwrap Tags
Wrap another tag around:


tag.wrap<span>(</span>soup.new_tag<span>(</span><span>'div)) </span>
tag.wrap<span>(</span>soup.new_tag<span>(</span><span>'div)) </span>
tag.wrap(soup.new_tag('div))

Enter fullscreen mode Exit fullscreen mode

Unwrap its contents:


tag.unwrap<span>()</span>
tag.unwrap<span>()</span>
tag.unwrap()

Enter fullscreen mode Exit fullscreen mode

Modifying the parse tree is very useful for cleaning up scraped data or extracting the parts you need.

Outputting HTML
Input HTML:


<p>Hello World</p>
<p>Hello World</p>
<p>Hello World</p>

Enter fullscreen mode Exit fullscreen mode

Python:


print<span>(</span>soup.prettify<span>())</span>
<span># <p></span>
<span># Hello World</span>
<span># </p></span>
print<span>(</span>soup.prettify<span>())</span>
<span># <p></span>
<span># Hello World</span>
<span># </p></span>
print(soup.prettify())
# <p>
# Hello World
# </p>

Enter fullscreen mode Exit fullscreen mode

Integrating with Requests
Fetch a page:


import requests
res <span>=</span> requests.get<span>(</span><span>"<https://example.com>"</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>res.text, <span>'html.parser'</span><span>)</span>
import requests
res <span>=</span> requests.get<span>(</span><span>"<https://example.com>"</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>res.text, <span>'html.parser'</span><span>)</span>
import requests
res = requests.get("<https://example.com>")
soup = BeautifulSoup(res.text, 'html.parser')

Enter fullscreen mode Exit fullscreen mode

Parsing Only Parts of a Document
When dealing with large documents, you may want to parse only a fragment rather than the whole thing. BeautifulSoup allows for this using SoupStrainers.
There are a few ways to parse only parts of a document:
By CSS Selector
Parse just a selection matching a CSS selector:


from bs4 import SoupStrainer
only_tables <span>=</span> SoupStrainer<span>(</span><span>"table"</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>only_tables<span>)</span>
from bs4 import SoupStrainer
only_tables <span>=</span> SoupStrainer<span>(</span><span>"table"</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>only_tables<span>)</span>
from bs4 import SoupStrainer
only_tables = SoupStrainer("table")
soup = BeautifulSoup(doc, parse_only=only_tables)

Enter fullscreen mode Exit fullscreen mode

This will parse only the
tags from the document.
By Tag Name
Parse only specific tags:


only_divs <span>=</span> SoupStrainer<span>(</span><span>"div"</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>only_divs<span>)</span>
only_divs <span>=</span> SoupStrainer<span>(</span><span>"div"</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>only_divs<span>)</span>
only_divs = SoupStrainer("div")
soup = BeautifulSoup(doc, parse_only=only_divs)

Enter fullscreen mode Exit fullscreen mode

By Function
Pass a function to test if a tag should be parsed:


def is_short_string<span>(</span>string<span>)</span>:
  <span>return </span>len<span>(</span>string<span>)</span> < 20
only_short_strings <span>=</span> SoupStrainer<span>(</span><span>string</span><span>=</span>is_short_string<span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>only_short_strings<span>)</span>
def is_short_string<span>(</span>string<span>)</span>:
  <span>return </span>len<span>(</span>string<span>)</span> < 20

only_short_strings <span>=</span> SoupStrainer<span>(</span><span>string</span><span>=</span>is_short_string<span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>only_short_strings<span>)</span>
def is_short_string(string):
  return len(string) < 20

only_short_strings = SoupStrainer(string=is_short_string)
soup = BeautifulSoup(doc, parse_only=only_short_strings)

Enter fullscreen mode Exit fullscreen mode

This parses tags based on their text content.

By Attributes
Parse tags that contain specific attributes:


has_data_attr <span>=</span> SoupStrainer<span>(</span><span>attrs</span><span>={</span><span>"data-category"</span>: True<span>})</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>has_data_attr<span>)</span>
has_data_attr <span>=</span> SoupStrainer<span>(</span><span>attrs</span><span>={</span><span>"data-category"</span>: True<span>})</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>has_data_attr<span>)</span>
has_data_attr = SoupStrainer(attrs={"data-category": True})
soup = BeautifulSoup(doc, parse_only=has_data_attr)

Enter fullscreen mode Exit fullscreen mode

Multiple Conditions
You can combine multiple strainers:


strainer <span>=</span> SoupStrainer<span>(</span><span>"div"</span>, <span>id</span><span>=</span><span>"main"</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>strainer<span>)</span>
strainer <span>=</span> SoupStrainer<span>(</span><span>"div"</span>, <span>id</span><span>=</span><span>"main"</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>parse_only</span><span>=</span>strainer<span>)</span>
strainer = SoupStrainer("div", id="main")
soup = BeautifulSoup(doc, parse_only=strainer)

Enter fullscreen mode Exit fullscreen mode

This will parse only

Parsing only parts you need can help reduce memory usage and improve performance when scraping large documents.

Dealing with Encoding
When parsing documents, you may encounter encoding issues. Here are some ways to handle encoding:

Specify at Parse Time
Pass the from_encoding parameter when creating the BeautifulSoup object:


soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>from_encoding</span><span>=</span><span>'utf-8'</span><span>)</span>
soup <span>=</span> BeautifulSoup<span>(</span>doc, <span>from_encoding</span><span>=</span><span>'utf-8'</span><span>)</span>
soup = BeautifulSoup(doc, from_encoding='utf-8')

Enter fullscreen mode Exit fullscreen mode

This handles any decoding needed when initially parsing the document.

Encode Tag Contents
You can encode the contents of a tag:


tag.string.encode<span>(</span><span>"utf-8"</span><span>)</span>
tag.string.encode<span>(</span><span>"utf-8"</span><span>)</span>
tag.string.encode("utf-8")

Enter fullscreen mode Exit fullscreen mode

Use this when outputting tag strings.

Encode Entire Document
To encode the entire BeautifulSoup document:


soup.encode<span>(</span><span>"utf-8"</span><span>)</span>
soup.encode<span>(</span><span>"utf-8"</span><span>)</span>
soup.encode("utf-8")

Enter fullscreen mode Exit fullscreen mode

This returns a byte string with the encoded document.

Pretty Print with Encoding
Specify encoding when pretty printing output:


print<span>(</span>soup.prettify<span>(</span><span>encoder</span><span>=</span><span>"utf-8"</span><span>))</span>
print<span>(</span>soup.prettify<span>(</span><span>encoder</span><span>=</span><span>"utf-8"</span><span>))</span>
print(soup.prettify(encoder="utf-8"))

Enter fullscreen mode Exit fullscreen mode

Unicode Dammit
BeautifulSoup’s UnicodeDammit class can detect and convert incoming documents to Unicode:


from bs4 import UnicodeDammit
dammit <span>=</span> UnicodeDammit<span>(</span>doc<span>)</span>
soup <span>=</span> dammit.unicode_markup
from bs4 import UnicodeDammit
dammit <span>=</span> UnicodeDammit<span>(</span>doc<span>)</span>
soup <span>=</span> dammit.unicode_markup
from bs4 import UnicodeDammit
dammit = UnicodeDammit(doc)
soup = dammit.unicode_markup

Enter fullscreen mode Exit fullscreen mode

This converts even poorly encoded documents to Unicode.

Properly handling encoding ensures your scraped data is decoded and output correctly when using BeautifulSoup.

1. `find()`

Purpose: Find the first occurrence of a tag.
Usage: soup.find('tag_name', {attributes}, text=optional_text)

Example:


<span>first_div</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>div</span><span>'</span><span>)</span>
<span>p_with_class</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>p</span><span>'</span><span>,</span> <span>class_</span><span>=</span><span>'</span><span>example</span><span>'</span><span>)</span>
<span>a_tag</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>a</span><span>'</span><span>,</span> <span>href</span><span>=</span><span>'</span><span>/home</span><span>'</span><span>)</span>
<span>first_div</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>div</span><span>'</span><span>)</span>
<span>p_with_class</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>p</span><span>'</span><span>,</span> <span>class_</span><span>=</span><span>'</span><span>example</span><span>'</span><span>)</span>
<span>a_tag</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>a</span><span>'</span><span>,</span> <span>href</span><span>=</span><span>'</span><span>/home</span><span>'</span><span>)</span>
first_div = soup.find('div')
p_with_class = soup.find('p', class_='example')
a_tag = soup.find('a', href='/home')

2. `find_all()`

Purpose: Find all occurrences of a tag.
Usage: soup.find_all('tag_name', {attributes}, limit=number)

Example:


<span>all_p_tags</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>p</span><span>'</span><span>)</span>
<span>all_links</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>a</span><span>'</span><span>,</span> <span>class_</span><span>=</span><span>'</span><span>link</span><span>'</span><span>)</span>
<span>first_five_divs</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>div</span><span>'</span><span>,</span> <span>limit</span><span>=</span><span>5</span><span>)</span>
<span>all_p_tags</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>p</span><span>'</span><span>)</span>
<span>all_links</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>a</span><span>'</span><span>,</span> <span>class_</span><span>=</span><span>'</span><span>link</span><span>'</span><span>)</span>
<span>first_five_divs</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>div</span><span>'</span><span>,</span> <span>limit</span><span>=</span><span>5</span><span>)</span>
all_p_tags = soup.find_all('p')
all_links = soup.find_all('a', class_='link')
first_five_divs = soup.find_all('div', limit=5)

3. `select()`

Purpose: Find all tags matching a CSS selector.
Usage: soup.select('CSS_selector')

Example:


<span>divs_with_class</span> <span>=</span> <span>soup</span><span>.</span><span>select</span><span>(</span><span>'</span><span>div.example</span><span>'</span><span>)</span>
<span>links_in_divs</span> <span>=</span> <span>soup</span><span>.</span><span>select</span><span>(</span><span>'</span><span>div a</span><span>'</span><span>)</span>
<span>element_with_id</span> <span>=</span> <span>soup</span><span>.</span><span>select</span><span>(</span><span>'</span><span>#specific-id</span><span>'</span><span>)</span>
<span>divs_with_class</span> <span>=</span> <span>soup</span><span>.</span><span>select</span><span>(</span><span>'</span><span>div.example</span><span>'</span><span>)</span>
<span>links_in_divs</span> <span>=</span> <span>soup</span><span>.</span><span>select</span><span>(</span><span>'</span><span>div a</span><span>'</span><span>)</span>
<span>element_with_id</span> <span>=</span> <span>soup</span><span>.</span><span>select</span><span>(</span><span>'</span><span>#specific-id</span><span>'</span><span>)</span>
divs_with_class = soup.select('div.example')
links_in_divs = soup.select('div a')
element_with_id = soup.select('#specific-id')

4. `find_parents()` / `find_parent()`

Purpose: Find parent(s) of a tag.
Usage: soup.find_parent('tag_name') or soup.find_parents('tag_name')

Example:


<span>parent_div</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>span</span><span>'</span><span>).</span><span>find_parent</span><span>(</span><span>'</span><span>div</span><span>'</span><span>)</span>
<span>all_parents</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>span</span><span>'</span><span>).</span><span>find_parents</span><span>()</span>
<span>parent_div</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>span</span><span>'</span><span>).</span><span>find_parent</span><span>(</span><span>'</span><span>div</span><span>'</span><span>)</span>
<span>all_parents</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>span</span><span>'</span><span>).</span><span>find_parents</span><span>()</span>
parent_div = soup.find('span').find_parent('div')
all_parents = soup.find('span').find_parents()

5. `find_next_sibling()` / `find_previous_sibling()`

Purpose: Find the next or previous sibling of a tag.
Usage: soup.find_next_sibling('tag_name') or soup.find_previous_sibling('tag_name')

Example:


<span>next_sibling</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>div</span><span>'</span><span>).</span><span>find_next_sibling</span><span>()</span>
<span>prev_sibling</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>div</span><span>'</span><span>).</span><span>find_previous_sibling</span><span>()</span>
<span>next_sibling</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>div</span><span>'</span><span>).</span><span>find_next_sibling</span><span>()</span>
<span>prev_sibling</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>div</span><span>'</span><span>).</span><span>find_previous_sibling</span><span>()</span>
next_sibling = soup.find('div').find_next_sibling()
prev_sibling = soup.find('div').find_previous_sibling()

6. `find_all_next()` / `find_all_previous()`

Purpose: Find all tags after or before a specific tag.
Usage: soup.find_all_next('tag_name') or soup.find_all_previous('tag_name')

Example:


<span>next_p_tags</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>h1</span><span>'</span><span>).</span><span>find_all_next</span><span>(</span><span>'</span><span>p</span><span>'</span><span>)</span>
<span>previous_div_tags</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>h2</span><span>'</span><span>).</span><span>find_all_previous</span><span>(</span><span>'</span><span>div</span><span>'</span><span>)</span>
<span>next_p_tags</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>h1</span><span>'</span><span>).</span><span>find_all_next</span><span>(</span><span>'</span><span>p</span><span>'</span><span>)</span>
<span>previous_div_tags</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>h2</span><span>'</span><span>).</span><span>find_all_previous</span><span>(</span><span>'</span><span>div</span><span>'</span><span>)</span>
next_p_tags = soup.find('h1').find_all_next('p')
previous_div_tags = soup.find('h2').find_all_previous('div')

7. `select_one()`

Purpose: Find the first element matching a CSS selector.
Usage: soup.select_one('CSS_selector')

Example:


<span>first_div_container</span> <span>=</span> <span>soup</span><span>.</span><span>select_one</span><span>(</span><span>'</span><span>div.container</span><span>'</span><span>)</span>
<span>first_link_in_main</span> <span>=</span> <span>soup</span><span>.</span><span>select_one</span><span>(</span><span>'</span><span>#main a</span><span>'</span><span>)</span>
<span>first_div_container</span> <span>=</span> <span>soup</span><span>.</span><span>select_one</span><span>(</span><span>'</span><span>div.container</span><span>'</span><span>)</span>
<span>first_link_in_main</span> <span>=</span> <span>soup</span><span>.</span><span>select_one</span><span>(</span><span>'</span><span>#main a</span><span>'</span><span>)</span>
first_div_container = soup.select_one('div.container')
first_link_in_main = soup.select_one('#main a')

8. `find_next()` / `find_previous()`

Purpose: Find the next or previous element in the document.
Usage: soup.find_next('tag_name') or soup.find_previous('tag_name')

Example:


<span>next_p_tag</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>div</span><span>'</span><span>).</span><span>find_next</span><span>(</span><span>'</span><span>p</span><span>'</span><span>)</span>
<span>previous_div</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>p</span><span>'</span><span>).</span><span>find_previous</span><span>(</span><span>'</span><span>div</span><span>'</span><span>)</span>
<span>next_p_tag</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>div</span><span>'</span><span>).</span><span>find_next</span><span>(</span><span>'</span><span>p</span><span>'</span><span>)</span>
<span>previous_div</span> <span>=</span> <span>soup</span><span>.</span><span>find</span><span>(</span><span>'</span><span>p</span><span>'</span><span>).</span><span>find_previous</span><span>(</span><span>'</span><span>div</span><span>'</span><span>)</span>
next_p_tag = soup.find('div').find_next('p')
previous_div = soup.find('p').find_previous('div')

9. `find_all(string=True)`

Purpose: Find all occurrences of a specific string or text.
Usage: soup.find_all(string="text_to_find")

Example:


<span>python_mentions</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>string</span><span>=</span><span>"</span><span>Python</span><span>"</span><span>)</span>
<span>programming_mentions</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>string</span><span>=</span><span>lambda</span> <span>text</span><span>:</span> <span>"</span><span>Programming</span><span>"</span> <span>in</span> <span>text</span><span>)</span>
<span>python_mentions</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>string</span><span>=</span><span>"</span><span>Python</span><span>"</span><span>)</span>
<span>programming_mentions</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>string</span><span>=</span><span>lambda</span> <span>text</span><span>:</span> <span>"</span><span>Programming</span><span>"</span> <span>in</span> <span>text</span><span>)</span>
python_mentions = soup.find_all(string="Python")
programming_mentions = soup.find_all(string=lambda text: "Programming" in text)

10. `find_all(True)` (Find all tags)

Purpose: Find all tags in the document.
Usage: soup.find_all(True)

Example:


<span>all_tags</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>True</span><span>)</span>
<span>all_tags</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>True</span><span>)</span>
all_tags = soup.find_all(True)

Example Use Cases

Find all links on a page:


<span>links</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>a</span><span>'</span><span>,</span> <span>href</span><span>=</span><span>True</span><span>)</span>
<span>for</span> <span>link</span> <span>in</span> <span>links</span><span>:</span>
    <span>print</span><span>(</span><span>link</span><span>[</span><span>'</span><span>href</span><span>'</span><span>])</span>
<span>links</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>a</span><span>'</span><span>,</span> <span>href</span><span>=</span><span>True</span><span>)</span>
<span>for</span> <span>link</span> <span>in</span> <span>links</span><span>:</span>
    <span>print</span><span>(</span><span>link</span><span>[</span><span>'</span><span>href</span><span>'</span><span>])</span>
links = soup.find_all('a', href=True)
for link in links:
    print(link['href'])

Find all headings (h1 to h6):


<span>headings</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>([</span><span>'</span><span>h1</span><span>'</span><span>,</span> <span>'</span><span>h2</span><span>'</span><span>,</span> <span>'</span><span>h3</span><span>'</span><span>,</span> <span>'</span><span>h4</span><span>'</span><span>,</span> <span>'</span><span>h5</span><span>'</span><span>,</span> <span>'</span><span>h6</span><span>'</span><span>])</span>
<span>for</span> <span>heading</span> <span>in</span> <span>headings</span><span>:</span>
    <span>print</span><span>(</span><span>heading</span><span>.</span><span>get_text</span><span>())</span>
<span>headings</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>([</span><span>'</span><span>h1</span><span>'</span><span>,</span> <span>'</span><span>h2</span><span>'</span><span>,</span> <span>'</span><span>h3</span><span>'</span><span>,</span> <span>'</span><span>h4</span><span>'</span><span>,</span> <span>'</span><span>h5</span><span>'</span><span>,</span> <span>'</span><span>h6</span><span>'</span><span>])</span>
<span>for</span> <span>heading</span> <span>in</span> <span>headings</span><span>:</span>
    <span>print</span><span>(</span><span>heading</span><span>.</span><span>get_text</span><span>())</span>
headings = soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5', 'h6'])
for heading in headings:
    print(heading.get_text())

Extract text from a specific class using CSS selector:


<span>text_in_class</span> <span>=</span> <span>soup</span><span>.</span><span>select_one</span><span>(</span><span>'</span><span>.specific-class</span><span>'</span><span>).</span><span>get_text</span><span>()</span>
<span>text_in_class</span> <span>=</span> <span>soup</span><span>.</span><span>select_one</span><span>(</span><span>'</span><span>.specific-class</span><span>'</span><span>).</span><span>get_text</span><span>()</span>
text_in_class = soup.select_one('.specific-class').get_text()

Find all images on a page:


<span>images</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>img</span><span>'</span><span>)</span>
<span>for</span> <span>image</span> <span>in</span> <span>images</span><span>:</span>
    <span>print</span><span>(</span><span>image</span><span>[</span><span>'</span><span>src</span><span>'</span><span>])</span>
<span>images</span> <span>=</span> <span>soup</span><span>.</span><span>find_all</span><span>(</span><span>'</span><span>img</span><span>'</span><span>)</span>
<span>for</span> <span>image</span> <span>in</span> <span>images</span><span>:</span>
    <span>print</span><span>(</span><span>image</span><span>[</span><span>'</span><span>src</span><span>'</span><span>])</span>
images = soup.find_all('img')
for image in images:
    print(image['src'])

Notes

find() and find_all() are the go-to methods for finding elements based on tag names and attributes.
select() and select_one() are very powerful if you’re comfortable with CSS selectors.
Navigational methods like find_next(), find_previous(), and find_parents() help when you need to traverse through sibling and parent tags.
find_all(string=True) is useful when searching for specific text rather than tags.

Additional Methods:

find_all(True) – Finds all tags in the document, useful when you want to iterate over everything.
get_text() – Extracts the text from a tag, stripping away HTML tags.


python
# Example of extracting text:
text = soup.find('p').get_text()
python
# Example of extracting text:
text = soup.find('p').get_text()

python
# Example of extracting text:
text = soup.find('p').get_text()

Enter fullscreen mode Exit fullscreen mode

原文链接：BeautifulSoup Cheat Sheet Python

展开阅读全文

文章版权声明 1、本网站名称：拾光赋
2、本站永久网址：https://www.blogs.ink
3、本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ：805375623进行删除处理。
4、本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5、本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6、本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。

THE END