|
之前的文章我们介绍了 re 模块和 lxml 模块来做爬虫，本章我们再来看一个 bs4 模块来做爬虫。 和 lxml 一样，Beautiful Soup 也是一个HTML/XML的解析器，主要的功能也是如何解析和提取 HTML/XML 数据。 lxml 只会局部遍历，而Beautiful Soup 是基于HTML DOM的，会载入整个文档，解析整个DOM树，因此时间和内存开销都会大很多，所以性能要低于lxml。 BeautifulSoup 用来解析 HTML 比较简单，API非常人性化，支持CSS选择器、Python标准库中的HTML解析器，也支持 lxml 的 XML解析器。 Beautiful Soup 3 目前已经停止开发，推荐现在的项目使用Beautiful Soup 4。使用 pip 安装即可：pip install beautifulsoup4 官方文档：http://beautifulsoup./zh_CN/v4.4.0
| 抓取工具 | 速度 | 使用难度 | 安装难度 |
|---|
| 正则 | 最快 | 困难 | 无（内置） | | BeautifulSoup | 慢 | 最简单 | 简单 | | lxml | 快 | 简单 | 一般 |
首先必须要导入 bs4 库 1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17
18 # 打开本地 HTML 文件的方式来创建对象
19 # soup = BeautifulSoup(open('index.html'), "lxml")
20
21 # 格式化输出 soup 对象的内容
22 print(soup.prettify())
运行结果： 1 <html>
2 <body>
3 <div>
4 <ul>
5 <li class="item-0">
6 <a href="link1.html">
7 first item
8 </a>
9 </li>
10 <li class="item-1">
11 <a href="link2.html">
12 second item
13 </a>
14 </li>
15 <li class="item-inactive">
16 <a href="link3.html">
17 <span class="bold">
18 third item
19 </span>
20 </a>
21 </li>
22 <li class="item-1">
23 <a href="link4.html">
24 fourth item
25 </a>
26 </li>
27 <li class="item-0">
28 <a href="link5.html">
29 fifth item
30 </a>
31 </li>
32 </ul>
33 </div>
34 </body>
35 </html>
四大对象种类Beautiful Soup将复杂HTML文档转换成一个复杂的树形结构,每个节点都是Python对象,所有对象可以归纳为4种: Tag NavigableString BeautifulSoup Comment
1. TagTag 通俗点讲就是 HTML 中的一个个标签，例如： 1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17
18 print(soup.li) # <li class="item-0"><a href="link1.html">first item</a></li>
19 print(soup.a) # <a href="link1.html">first item</a>
20 print(soup.span) # <span class="bold">third item</span>
21 print(soup.p) # None
22 print(type(soup.li)) # <class 'bs4.element.Tag'>
我们可以利用 soup 加标签名轻松地获取这些标签的内容，这些对象的类型是bs4.element.Tag。但是注意，它查找的是在所有内容中的第一个符合要求的标签。如果要查询所有的标签，后面会进行介绍。对于 Tag，它有两个重要的属性，是 name 和 attrs
1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17
18 print(soup.li.attrs) # {'class': ['item-0']}
19 print(soup.li["class"]) # ['item-0']
20 print(soup.li.get('class')) # ['item-0']
21
22 print(soup.li) # <li class="item-0"><a href="link1.html">first item</a></li>
23 soup.li["class"] = "newClass" # 可以对这些属性和内容等等进行修改
24 print(soup.li) # <li class="newClass"><a href="link1.html">first item</a></li>
25
26 del soup.li['class'] # 还可以对这个属性进行删除
27 print(soup.li) # <li><a href="link1.html">first item</a></li>
2. NavigableString既然我们已经得到了标签的内容，那么问题来了，我们要想获取标签内部的文字怎么办呢？很简单，用 .string 即可，例如 1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17
18 print(soup.li.string) # first item
19 print(soup.a.string) # first item
20 print(soup.span.string) # third item
21 # print(soup.p.string) # AttributeError: 'NoneType' object has no attribute 'string'
22 print(type(soup.li.string)) # <class 'bs4.element.NavigableString'> 3. BeautifulSoupBeautifulSoup 对象表示的是一个文档的内容。大部分时候,可以把它当作 Tag 对象，是一个特殊的 Tag，我们可以分别获取它的类型，名称，以及属性来感受一下
1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17
18 print(soup.name) # [document]
19 print(soup.attrs) # {}, 文档本身的属性为空
20 print(type(soup.name)) # <class 'str'>4. CommentComment 对象是一个特殊类型的 NavigableString 对象，其输出的内容不包括注释符号。 1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <a class="sister" href="http:///elsie" id="link1"><!-- Elsie --></a>
6 </div>
7 """
8
9 # 创建 Beautiful Soup 对象
10 soup = BeautifulSoup(html, "lxml")
11
12 print(soup.a) # <a class="sister" href="http:///elsie" id="link1"><!-- Elsie --></a>
13 print(soup.a.string) # Elsie
14 print(type(soup.a.string)) # <class 'bs4.element.Comment'>
a 标签里的内容实际上是注释，但是如果我们利用 .string 来输出它的内容时，注释符号已经去掉了。 遍历文档树1. 直接子节点 ：.contents .children 属性.contenttag 的 .content 属性可以将tag的子节点以列表的方式输出，输出方式为列表，我们可以用列表索引来获取它的某一个元素
1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17
18 print(soup.li.contents) # [<a href="link1.html">first item</a>]
19 print(soup.li.contents[0]) # <a href="link1.html">first item</a>
.children它返回的不是一个 list，不过我们可以通过遍历获取所有子节点。 我们打印输出 .children 看一下，可以发现它是一个 list 生成器对象
1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17
18 print(soup.ul.children) # <list_iterator object at 0x106388a20>
19 for child in soup.ul.children:
20 print(child)
输出结果：
1 <li class="item-0"><a href="link1.html">first item</a></li>
2
3
4 <li class="item-1"><a href="link2.html">second item</a></li>
5
6
7 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
8
9
10 <li class="item-1"><a href="link4.html">fourth item</a></li>
11
12
13 <li class="item-0"><a href="link5.html">fifth item</a></li>
2. 所有子孙节点: .descendants 属性.contents 和 .children 属性仅包含tag的直接子节点，.descendants 属性可以对所有tag的子孙节点进行递归循环，和 children类似，我们也需要遍历获取其中的内容。 1 for child in soup.ul.descendants:2 print(child) 运行结果：
1 <li class="item-0"><a href="link1.html">first item</a></li>
2 <a href="link1.html">first item</a>
3 first item
4
5
6 <li class="item-1"><a href="link2.html">second item</a></li>
7 <a href="link2.html">second item</a> 8 second item
9
10
11 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
12 <a href="link3.html"><span class="bold">third item</span></a>
13 <span class="bold">third item</span>
14 third item
15
16
17 <li class="item-1"><a href="link4.html">fourth item</a></li>
18 <a href="link4.html">fourth item</a>
19 fourth item
20
21
22 <li class="item-0"><a href="link5.html">fifth item</a></li>
23 <a href="link5.html">fifth item</a>
24 fifth item
搜索文档树1.find_all(name, attrs, recursive, text, **kwargs)1）name 参数name 参数可以查找所有名字为 name 的 tag，字符串对象会被自动忽略掉 A.传字符串最简单的过滤器是字符串.在搜索方法中传入一个字符串参数,Beautiful Soup会查找与字符串完整匹配的内容,下面的例子用于查找文档中所有的<span>标签: 1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17 print(soup.find_all('span')) # [<span class="bold">third item</span>]
B.传正则表达式如果传入正则表达式作为参数,Beautiful Soup会通过正则表达式的 match() 来匹配内容.下面例子中找出所有以 s 开头的标签,这表示 <span> 标签都应该被找到
1 from bs4 import BeautifulSoup
2 import re
3 4 html = """
5 <div>
6 <ul>
7 <li class="item-0"><a href="link1.html">first item</a></li>
8 <li class="item-1"><a href="link2.html">second item</a></li>
9 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
10 <li class="item-1"><a href="link4.html">fourth item</a></li>
11 <li class="item-0"><a href="link5.html">fifth item</a></li>
12 </ul>
13 </div>
14 """
15
16 # 创建 Beautiful Soup 对象
17 soup = BeautifulSoup(html, "lxml")
18 for tag in soup.find_all(re.compile("^s")):
19 print(tag)
20 # <span class="bold">third item</span>
C.传列表如果传入列表参数,Beautiful Soup会将与列表中任一元素匹配的内容返回.下面代码找到文档中所有 <a> 标签和 <span> 标签:
1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17 print(soup.find_all(["a", "span"]))
18 # [<a href="link1.html">first item</a>, <a href="link2.html">second item</a>, <a href="link3.html"><span class="bold">third item</span></a>, <span class="bold">third item</span>, <a href="link4.html">fourth item</a>, <a href="link5.html">fifth item</a>]
2）keyword 参数
1 from bs4 import BeautifulSoup
2
3 html = """
4 <div>
5 <ul>
6 <li class="item-0"><a href="link1.html">first item</a></li>
7 <li class="item-1"><a href="link2.html">second item</a></li>
8 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
9 <li class="item-1"><a href="link4.html">fourth item</a></li>
10 <li class="item-0"><a href="link5.html">fifth item</a></li>
11 </ul>
12 </div>
13 """
14
15 # 创建 Beautiful Soup 对象
16 soup = BeautifulSoup(html, "lxml")
17 print(soup.find_all(href='link1.html')) # [<a href="link1.html">first item</a>]
3）text 参数通过 text 参数可以搜搜文档中的字符串内容，与 name 参数的可选值一样, text 参数接受 字符串 , 正则表达式 , 列表 1 from bs4 import BeautifulSoup
2 import re
3
4 html = """
5 <div>
6 <ul>
7 <li class="item-0"><a href="link1.html">first item</a></li>
8 <li class="item-1"><a href="link2.html">second item</a></li>
9 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
10 <li class="item-1"><a href="link4.html">fourth item</a></li>
11 <li class="item-0"><a href="link5.html">fifth item</a></li>
12 </ul>
13 </div>
14 """
15
16 # 创建 Beautiful Soup 对象
17 soup = BeautifulSoup(html, "lxml")
18 print(soup.find_all(text="first item")) # ['first item']
19 print(soup.find_all(text=["first item", "second item"])) # ['first item', 'second item']
20 print(soup.find_all(text=re.compile("item"))) # ['first item', 'second item', 'third item', 'fourth item', 'fifth item']CSS选择器这就是另一种与 find_all 方法有异曲同工之妙的查找方法. （1）通过标签名查找
1 from bs4 import BeautifulSoup
2 import re
3
4 html = """
5 <div>
6 <ul>
7 <li class="item-0"><a href="link1.html">first item</a></li>
8 <li class="item-1"><a href="link2.html">second item</a></li>
9 <li class="item-inactive"><a href="link3.html"><span class="bold">third item</span></a></li>
10 <li class="item-1"><a href="link4.html">fourth item</a></li>
11 <li class="item-0"><a href="link5.html">fifth item</a></li>
12 </ul>
13 </div>
14 """
15
16 # 创建 Beautiful Soup 对象
17 soup = BeautifulSoup(html, "lxml")
18 print(soup.select('span')) # [<span class="bold">third item</span>]（2）通过类名查找1 print(soup.select('.item-0'))
2 # [<li class="item-0"><a href="link1.html">first item</a></li>, <li class="item-0"><a href="link5.html">fifth item</a></li>]（3）通过 id 名查找print(soup.select('#item-0')) # []（4）组合查找1 print(soup.select('li.item-0'))
2 # [<li class="item-0"><a href="link1.html">first item</a></li>, <li class="item-0"><a href="link5.html">fifth item</a></li>]
3 print(soup.select('li.item-0>a'))
4 # [<a href="link1.html">first item</a>, <a href="link5.html">fifth item</a>]（5）属性查找1 print(soup.select('a[href="link1.html"]')) # [<a href="link1.html">first item</a>] (6) 获取内容
1 for text in soup.select('li'):
2 print(text.get_text())
3 """
4 first item
5 second item
6 third item
7 fourth item
8 fifth item
9 """ |
