Web Crawler Get the User-Agent

Posted on | Edited on | In | Views:

How to Get User-Agent for your Crawler

There are several common ways to get the user-agent and use it to scrape the website.

  1. about:version in your browser, and check the User-Agent.

  2. Use the inspect tool within your browser. Network -> refresh your page -> find the current page -> Headers -> check the User-Agent

  3. install the fake_agent in your computer and import it for using in Python

    1
    pip install fake_useragent

    When using:

    1
    2
    3
    4
    5
    from fake_useragent import UserAgent
    # import random

    fake_ua=UserAgent()
    headers={'User-Agent':fake_ua.random}
  1. Google to find some User-Agent, sush as User-Agent 汇总

Reference

  1. CSDN blog: 爬虫之UserAgent的获得方法
  2. fake_useragent doc