|
这是一个Python脚本，用于从名为“xiezhen”的网站下载图片。它使用请求库向网站发送 HTTP 请求，并使用 lxml 库解析响应的 HTML 内容。
该脚本定义了两个函数：download_image和process_page。该download_image函数以一个URL和一个文件路径为参数，向URL发送请求获取图片内容，并将内容保存到文件路径中。该process_page函数以页码为参数，向网站发送请求获取该页面的HTML内容，解析内容提取图片页面的URL，然后为每个图片URL调用该函数下载对应的download_image图片图像。
该脚本还使用了concurrent.futures模块创建了一个线程池，这样可以并行发送多个请求和下载图片，可以提高下载速度。
在主程序块中，脚本创建一个线程池，并向该池提交任务以从网站的所有页面下载图像。最后，脚本在退出之前等待所有任务完成。 import timeimport requestsfrom lxml import etreeimport osimport concurrent.futures def download_image(url, img_path): headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'} response = requests.get(url, headers=headers) img_name = url.split('/')[-1] with open(os.path.join(img_path, img_name), 'wb') as f: f.write(response.content) print(f'图片：{img_path}' + '/' + f'{img_name}下载完成！') def process_page(page): url = f'https://www./page/{page}' headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'} response = requests.get(url, headers=headers) html = etree.HTML(response.content) mail_url = html.xpath('//div[@class='excerpts']/article/a/@href') for url in mail_url: response = requests.get(url, headers=headers) html = etree.HTML(response.content) sub_url = html.xpath('//article/p/img') img_title = html.xpath('//title/text()')[0].split('-')[0] img_path = f'J:/xiezhen/{img_title}' if not os.path.exists(img_path): os.makedirs(img_path) with concurrent.futures.ThreadPoolExecutor() as executor: futures = [] for s_url in sub_url: img_url = s_url.attrib['src'] futures.append(executor.submit(download_image, img_url, img_path)) for future in concurrent.futures.as_completed(futures): pass time.sleep(0.5) if __name__ == '__main__': with concurrent.futures.ThreadPoolExecutor() as executor: futures = [] for page in range(1, 573): futures.append(executor.submit(process_page, page)) for future in concurrent.futures.as_completed(futures): pass
|
