Python爬虫框架详解介绍

文章目录

前言

Python 提供了多种强大的爬虫框架，适用于不同场景和需求。以下是主流框架的详细介绍及对比分析：

一、Requests + BeautifulSoup（基础组合）

特点：

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

# 提取所有链接
links = [a['href'] for a in soup.find_all('a', href=True)]

# 提取特定元素
title = soup.find('h1').text

二、Scrapy（高级框架）

特点：

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"
    start_urls = ["https://example.com"]

    def parse(self, response):
        # 提取数据
        for item in response.css('div.item'):
            yield {
                'title': item.css('h2::text').get(),
                'link': item.css('a::attr(href)').get(),
            }
        
        # 跟进链接
        next_page = response.css('a.next-page::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)
运行命令：
bash
scrapy startproject myproject
scrapy genspider example example.com
scrapy crawl example -o items.json

三、PySpider（可视化爬虫）

特点：

pip install pyspider
pyspider all

示例代码：

from pyspider.libs.base_handler import *

class Handler(BaseHandler):
    @every(minutes=24 * 60)
    def on_start(self):
        self.crawl('https://example.com', callback=self.index_page)
    
    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        for each in response.doc('a[href^="http"]').items():
            self.crawl(each.attr.href, callback=self.detail_page)
    
    def detail_page(self, response):
        return {
            "url": response.url,
            "title": response.doc('title').text(),
        }
        

四、Selenium（浏览器自动化）

特点：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service

# 设置 ChromeDriver 路径
service = Service('/path/to/chromedriver')
driver = webdriver.Chrome(service=service)

driver.get('https://example.com')

# 等待元素加载
element = driver.find_element(By.CSS_SELECTOR, 'button.submit')
element.click()

# 提取数据
data = driver.find_element(By.CSS_SELECTOR, 'div.result').text
driver.quit()

五、Playwright（新一代浏览器自动化）

特点：

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto('https://example.com')
    
    # 填写表单
    page.fill('input[name="search"]', 'Python')
    page.click('button[type="submit"]')
    
    # 等待结果
    page.wait_for_selector('div.result-item')
    
    # 提取数据
    results = page.query_selector_all('div.result-item')
    for result in results:
        print(result.text_content())
    
    browser.close()

作者：英英_