目录一、浏览器的一般设置二、打开网页并获取网页源码的方式1、基于requests库2、基于urlib库3、基于selenium三、HTML解析1、BeautifulSoup2、Selenium动态渲染爬虫：模拟动态操作网页，加载JS(webdriver)1) 8种find_element定位元素的方法：2）frame、window切换：driver.switch_to.3）一些特殊情况：3、scrapy框架四、反爬虫操作五、 一些常见的异常处理六、url接口调用问题——get、post操作1、get请求2、post请求fromseleniumimportwebdriverfromselenium.webdriver.common.desired_capabilitiesimportDesiredCapabilitiesfromselenium.webdriver.common.action_chainsimportActionChainsimporturllib.parseimporturllib.requestfrombs4importBeautifulSoup一、浏览器的一般设置对于Chrome：li_user_agent=['Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11','Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1','Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16','Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)']chromedriver='D:\cqf_软件\python36\chromedriver.exe'#修改，将Chromedriver放在Python安装路径下的Scripts文件中，因此在启动Chrome时不再指定Chromedriver（写明其存放路径）chome_options=webdriver.ChromeOptions()chome_options.add_argument(('--proxy-server=http://'+proxy_ip[0]+':'+proxy_ip[1]))#添加启动参数（代理IP）chome_options.add_argument('user-agent='.format(random.choice(li_user_agent)))# 添加报头chome_options.add_argument('--headless')# 无头浏览器模式chome_options.add_argument('--disable-gpu')prefs_images={'profile.managed_default_content_settings.images':2}chome_options.add_experimental_option("prefs",prefs_images)#（不加载图片）browser=webdriver.Chrome(chromedriver,chrome_options=chome_options)对于PhantomJS：dcap=dict(DesiredCapabilities.PHANTOMJS)dcap["phantomjs.page.settings.userAgent"]=(random.choice(li_user_agent))#设置浏览器报头dcap["phantomjs.page.settings.loadImages"]=False#不加载图片service_args=['--proxy=%s'%proxy_ip,'--proxy-type=http']设置使用代理IP browser=webdriver.PhantomJS(desired_capabilities=dcap,service_args=service_args)二、打开网页并获取网页源码的方式1、基于requests库importrequests url='https://book.douban.com/'headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3880.400 QQBrowser/10.8.4554.400 '}request=requests.get(url,headers=headers)request.encoding='utf-8'print('text:',request.text)# text返回的是Unicode型的数据。结果为空时，说明目标网站设置了反爬虫，添加headers参数一般能解决