52  Beautiful Soup (Basic)

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')

print(soup.prettify())
<html>
 <head>
  <title>
   The Dormouse's story
  </title>
 </head>
 <body>
  <p class="title">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    Elsie
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>

52.1 Basic Usage

52.1.1 Title

soup.title
<title>The Dormouse's story</title>
soup.title.name
'title'
soup.title.string
"The Dormouse's story"
soup.title.parent.name
'head'

52.1.2 Paragraph

soup.p
<p class="title"><b>The Dormouse's story</b></p>
soup.p['class']
['title']

52.2 Tasks

52.2.1 Extracting all the URLs

Extracting all the URLs found within a page’s <a> tags:

for link in soup.find_all('a'):
    print(link.get('href'))
http://example.com/elsie
http://example.com/lacie
http://example.com/tillie
[link.get("href") for link in soup.find_all('a')]
['http://example.com/elsie',
 'http://example.com/lacie',
 'http://example.com/tillie']
[link.name for link in soup.find_all('a')]
['a', 'a', 'a']
[link.string for link in soup.find_all('a')]
['Elsie', 'Lacie', 'Tillie']

52.2.2 Extract all Text

Extracting all the text from a page:

soup.get_text()
"The Dormouse's story\n\nThe Dormouse's story\nOnce upon a time there were three little sisters; and their names were\nElsie,\nLacie and\nTillie;\nand they lived at the bottom of a well.\n...\n"